Testing for NSD!

[norb]

Testing for NSD is hard. It's lots of hours of running the same scenario over and over. It's not even really playing, it's looking for bugs. It is NOT designing, the game already has designers. It is NOT critiquing the artwork, it is only finding bugs with the artwork. There is a difference. You are going to get a lot of crap as a tester, because no one likes to hear that the made a mistake, especially me. I'm a perfectionist and most of the bugs are going to be my mistakes. If you are not doing your job, you are going to be kicked off the team. If you don't have cable, fios, or fast dsl, don't even apply. You will have to download new versions often from our private ftp server. You will have to be able to play multiplayer over a good connection. You will have to have lots of time to run the game. You will be assigned things to test and you will be expected to run those things through their paces. That means running them over and over again until you are willing to risk your reputation that they are as perfect and as bug free as humans can be expected to make them. We will then fix something and you will have to do it all again... and again... and again... until you know the scenario or open play or multiplayer until you know it better than the designer. You will have to have a decent computer, if you can't run the game you can't test. You will be required to sign an NDA and snail mail it to NSD. That says that you can't talk about the game unless we give you permission. You can't post screen shots. You can't give out the test version to anyone. You can install on multiple computers in your home and get help from your family to test MP. You will have to be able to take direction, Jim, Oliver, and myself will be giving the orders and you'll be expected to follow them. You will have to put your pride on hold, as we are under a lot of pressure and although we are all respectful to each other, sometimes we are curt. So if you're overly sensitive, not a good idea to apply. If you only want to test the big name battles, take a hike. We'll decide what everyone tests. Also, if we don't know you, you probably won't get on the team. It is doubtful that we would accept anyone that can't be vouched for, sorry, but we've been burned in the past and don't want to take chances.

What is testing? It is finding bugs. But not only that, you MUST be able to RECREATE those bugs. If I cannot recreate a bug, I can't fix it. That means that you run with autosave on, then when you see a bug you immediately stop the game, rename the autosave and load it. Then start from that save and document the EXACT steps to recreate the bug, telling exactly where to look and what to look for. You then must go into our online bug database and be able to document the bug in a clear and concise manner so that it can be worked on. You also need to be familiar with what's in the database to make sure that it's not been documented already. But when in doubt, just put it in. Anything that you think is a bug, you should document. Then one of the team will decided if it's really a bug. Sometimes you'll have to fight for what you believe in, say your bug gets shot down, but you are sure that you are right. If you make a good case, you might be right. Might have to go to someone else on the team to get them to listen. It's a tough job. Then you take your save, name it to match the bug number and upload it to our server. Then when it's marked as fixed, you have to go in and retest it to make sure that it really is fixed. It's a process, but it works.

Still want to test for us Save your sanity and don't apply if you have any doubts.

What do you get? You get to be on the inside. You get access to our server and you get to preview the game and get some of your input in the game. You get your name in the game. You might even get to write a short line in the credits. You might get your name on the scenarios that you are responsible for, have to see how this would work. You get no money. You do get access to a private testing forum and a testing mailing list and our bug database. You'll get a forum title. You will be expected to be around to answer questions once the game is out. You will be our first line of forum support.

No we are not ready yet, but I wanted to make sure that everyone had time to consider the responsibilities before we open up for volunteers.

[estabu2]

The truth rears its ugly head!!! Can we send you a PM with some questions or is there another way you would prefer to be contacted?

[JC Edwards]

I'm all in when your ready Norb.

[estabu2]

JC Edwards wrote:

I'm all in when your ready Norb.

I 2nd that. 'crosses fingers'

[ironsight]

Norb wrote:
You are going to get a lot of crap as a tester, because no one likes to hear that the made a mistake

For any of you guys planning on being a tester, take heed of that statement! Having been on both sides of the fence, a Design Engineer and a Test Engineer of primarily electronics hardware with some software thrown in, FWIW i just want to re-enforce what Norb said having been there done all that myself.

A.) Some Designers will argue til they're blue in the face that they're right and you're wrong. A good tester needs detailed documentation, accurate reproducable hard test data and most important, dogged persistance. Some designers seriously welcome bug reports as they really want their design to be as perfect as humanly possible. But a lot of designers unfortunately let ego get in the way. And then there's potential project deadline delays due to fixing bugs which might impact a designer's bonus, promotion or raise. There can be an adversarial relationship between designers and testers..just beware of that.

B.) Test Engineering is generallly not for the timid or faint of heart especially if you're susceptable to getting your feelings hurt.

C.) If for some reason a bug is NOT taken seriously and fixed, document it all in some official capacity to cover your ass for later on when the product starts dropping like flies in the field. I've seen a lot of bugs knowingly released per time-to-market concerns with the intent of fixing it later in an update revision. Thats OK as frustrating as it might be to a tester; you documented it, it was noted, you did your job, get over it and let the boss be boss.

D.) Random hard to reproduce bugs are difficult at best and are usually overlooked as real bugs only to show up later in the field. Don't ignore them cause they're not 100% reproducable! The only thing one can do barring an automated test system which usually will nail and allow fixing of random bugs...is to keep doing the same thing over and over, document the attempts, conditions and the percentage rate the bug appears. Hopefully a pattern will emerge.
Sometimes changing the software settings/conditions one parameter at a time will allow the bug to surface repeatedly. Don't overlook PC, GPU or other environmental parameters.

Good Luck to you all! Looking forward to a bugless release of the new game.
:

[norb]

Good stuff ironsight! Most of our bugs are random, as the AI is written to be unpredictable. Our bug tracking system holds all the bugs, even the ones turned down will have the documentation as to why. Jim and myself have been through this before and we know what it takes. We work very hard to not let our ego get in the way of the goal, a bug free release. It's hard sometimes, especially for me when the bug comes back NOT FIXED for the fourth time. The goal of everyone here is to have a bug free product, so as long as we don't lose site of that, we'll all work well together.

[Jim]

Everything that Norb says is true except he left out the hardest part of being a beta tester. The hardest part is actually having the discipline to pull yourself out of commanding the battle to go back and see if that odd behavior happens again. Proving Murphy's law applies to beta testing, bugs usually show up just when you are making that critical move to get on the enemies flank and rout their battle line from end to end.

The real reward is knowing that you helped shape the next generation of combat sim.

If you have a question you don't want to post in this thread, send me a PM. I don't check here every night, but I will get back to you.

[UglyElmo]

The worst part with being a "good" beta tester is you must try to break the game. Sometimes you have to do things over and over, with slight variations and become anal-retentive about writing down the details of each test.

For example: When I was a member of the patch-testing team for Creative Assembly's first two games of the Total War series (Shogun and Medieval 1) I remember having to make a custom map to test the shooting distances of the various arrow and gunpowder units. I would move another unit towards the unit I was testing and write down the distance the approaching unit was at when it first began taking fire from the test unit. Not only that, I also had to test how different weather conditions affected these distances, as well as line-of-sight issues. I also had to test given the different firing modes of the units (fire-at-will, manual targeting etc.)

All of the above was just a small part of the testing which needed to be done, and it took days and days of work (and believe me it is work) to get through it. I found an issue with units firing through hills/trees as well as problems with when napalm units would begin firing (the units skirmish (fall back/retreat) distance was larger than it's firing distance, so the units would run before ever firing. The only way you could get them to fire was to place them in "hold position" mode.

The napalm unit problems were fixed, but CA decided to leave the problem with units firing through hills/trees alone. It was frustrating to be aware of a problem, which was not going to be addressed, but it was part of the job.

Not everything you find will be addressed and you have no rights in demanding they be fixed, since it is not what you were selected to do. I found out later the reason CA decided to not put in resources into solving the firing-through-hills problem was the fact they felt most players would never notice the problem and it was going to be very difficult to solve given the current engine's capabilities.

I still enjoyed the process since I was helping the game I enjoyed to become better . . . not perfect . . . but better.

Lastly, the NDA Non-Disclosure Agreement will not allow you to speak of anything you are currently doing with anyone outside of the development/testing team. Do not take it lightly, and always be aware of your discussions in the forums. You will have to sometimes bite your tongue when you see someone complaining about an issue, which you know is being addressed or which shows their blatant misunderstanding of the details.

If you are still willing to work under the above conditions then "Bless you" and I hope you are selected to work on the team.

Myself: I would be willing to work on testing any MP features and MP playability issues, since that is my main interest. If you need someone to help out with MP testing or provide some ideas for the lobby, MP gameplay, etc . . please feel free to PM me with any questions. I loved the Take Command series and think this game would easily surpass those if the MP side of the game is implemented successfully.

Good luck either way. I will check back on a regular basis to keep up with the development of this game. I am excited to see a game which may provide the excitement of my first online gaming experiences with Sid Meier's Gettysburg! and the old MPlayer community. :laugh:

[GShock]

I will evidently apply when the time comes.

[vest]

Im new hear but not new to multiplayer games or gaming in general, with that said, have you fella's heard of Ventrilo? it is a voice server that you can purchase monthly and have verious sizes. all you need is to download the program (google search) use a mic and headphones and you are able to talk to anyone on that voice server. I have had one for years. its only like less than 5 bucks a month for the person who owns the server and you can invite as many people as you have slots on the server to join it for free. Hit me up if you want to download the program and test it out on my voice server. ill help you from there. Also there is a program called xfire which is a very simple (free) way to communicate with other of a group. its like a pm system but it also has many other features like your own (free) clan website and stuff like that. Im sure there is more i could tell you about if needed.