Ya know, I am normally a person who very much lives by the golden rule, live and let live, easy going. But something happened to me recently that made me want to go find someone, kidnap them and create a video as a warning to anyone else that would create the insidious type of extortion that is the self installing, self propagating “anti spyware” maleware.

(actually I have a very good idea of what I wanted to do, but it was so graphically violent I was afraid I’d get a visit from the FBI the day after posting it)

This blog entry is in two parts:

The Second part details (in great detail) how to get rid of malware that installs itself onto your computer, by hand. This is not a Terse how to, but provides more detail on to what kinds of things to look for and how. Also an attempt to document a specific technique that can be used on any type of software. If this is what you are interested scroll down to “How To”.

First is a personal reasoning of why malicious malware authors are guilty of organized blackmail, extortion, racketeering, trespassing, and another number of other offenses, and should be prosecuted with all the zeal and resources as any other organized crime. What they do is no different from a gang shaking down locals for protection money.

What act am I talking about? A tracking cookie was hijacked and used to install a program called “Antimalware Doctor” on one of the windows vista systems in my house. I lost several hours of my life removing this, this, abomination, from my system. Fortunately for me, this type of thing has happened before in my house. Usually to one of my kids who would pick up something from one of the free game websites (I tracked one back to newgrounds.com once, before they implemented security changes, now newgrounds is one of the safest places to play free games).

This is a type of malware that installs itself, and tries to disguise itself as windows defender, and other legitimate anti virus software.

In this case it installed a trey icon that popped up messages saying the system is infected.

This particular version, “Antimalware Doctor” is particularly nasty, it manages to insert iself as a legimit virus scanner (in this case I think it was using a debugging feature).

After a few minuites of operation, “Antimalware Doctor” Starts flagging _every_ process as potentially infected, preventing any new process from starting. It also seemed to detect when taskmgr.exe or regedit.exe where executed. I discovered that if askmgr.exe or regedit.exe was the first thing I ran (or put it in the start menu) then “Antimalware Doctor” couldn’t prevent them from executing. But anything else launched after that would fail to execute. (admittedly, I did not try running them from the command prompt, so the mechanism could have been to intercept windows explorer)

Another thing this nasty bugger did to the system was set up a system wide proxy so that Internet Explorer, Firefox, Chrome, and any other browser could not get to the internet. The proxy actually directed all internet traffic to go through 127.0.0.1 (For those of you who have never seen this before, this is an internal address of your own computer) Malware doctor created a web page on the local system that read “this page is infected”. A hapless user get’s this page NO MATTER WHERE they go. www.google.com and www.microsoft.com came up as infected.

One more bit of text before I get to the good stuff. Who ever wrote this nasty thing, and those who have written it’s brethren. Let it be known that the authors, and distributors of this thing are committing extortion and blackmail. In the truest sense of their definitions.

I submit to you one of the legal definitions of extortion and blackmail:

The obtaining of property from another induced by wrongful use of actual or threatened force, violence, or fear, or under color of official right.

The crime involving a threat for purposes of compelling a person to do an act against his or her will, or for purposes of taking the person’s money or property.

The goal of “Antimalware Doctor” is to get you to go to a website and pay money to get rid of it. Simple as that. They did something to me, I didn’t want done, and they will keep doing it to me. Part of what they are doing is demanding money from me, another thing they are doing is preventing me from earning a living by effectively rendering the tools of my trade inoperable until I pay a fee. (I am hoping that some attorney general will take up a class action case and go after these racketeers). Yes I honestly believe the authors of “Malware Doctor” are guilty of crimes that are covered under the US RICO act. I would greatly enjoy discussing this with any organization willing to peruse legal action against the authors of “Antimalware Doctor”.

I felt like someone broke into my house, filled it with boxes, then demanded I pay a moving company. without even promising to take the boxes away.

Barring legal action, If you find yourself infected with “Antimalware Doctor” here are the steps I used to get rid of it. Although this process, and registry entries are for “Antimalware Doctor” this technique should work for most forms of malware.

How To

(author’s note: I was not able to use “revo-uninstaller” because it was not on this particular system. I suspect revo’s “hunter” mode would have been able to nuke this guy. I have a lot of respect for the authors of Revo Uninstaller and highly recommend it to anyone who does software development, plays a lot of games, or does casual experimentation on a system. Disclaimer: I do not know the authors of Revo, I have never worked with them (to the best of my knowledge). But if I ever did meet them I’d like to buy them a beer)

In general the procedure I use is:

#1 find the executables in the file system and turn off all permissions

#2 rename the executable, and the directories they are in

#3 Boot the system into “safe mode”

(At this point it might be easier just restoring the system to the last “Save Point”, but sometimes that does not work)

#4 search for the executables in the registry and document GUIDs and other keywords

#5 search the registry for those GUIDs and other keywords, and remove them

#6 Finally, delete the registry entries for the executables, and the executables themselves.

#7 Fix the bogus “Proxy” settings

#1 Find the executable

Ctrl+alt+delete and launch task manager.

(Also note, you need local administrative privileges to do most of this work)

Normal applications will appear under the “applications” tab. You will not find it there, you need to go to processes, and enable viewing of all processes.

All malware I have encountered use random or obscure executable names. For Antimalware Doctor, the executables did not have a description, a sure sine of something you don’t want running.

In this case the executable names are

newsecureapp70700.exe

and

vwmpqicshdw.exe

First, be aware that new versions of windows allow you to control the permissions an executable has.

How here is a handy trick. In windows task manager right click on the executable name. You should see “open file location” on the top. click that.

A file browser window will open containing the offending files.

your next goal is to find any executable or extension and remove all of it’s permissions.

There may be several executables and .dll files. Antimalware Doctorput it’s own .dlls and executables in a directory by itself, so they where easy to find. In this case you want to change the permissions on everything in the directory.

Some malware will put there executables in a system directory alongside a bunch of important operating system files. In that case you need to be really careful what you change, rename and delete.

when you right clicked on the executable (in windows task manager) an selected “Open file location” the executable should be selected in the file browser.

Right click on the offending executable.

select “Properties” (last menu item on the bottom)

(also note the title of the window will have the executable’s name, quickly double check you are changing the properties for the right executable)

Click on the “Security” Tab at the top of the properties window

There are two “panes” in this window. The top pane has a list of groups and users, the bottom pane has the permissions.

Your goal is set ALL permissions for ALL users to Deny

First you will need to click the Edit button, and click Continue in the popup (Vista, windows 7 may be different)

Click on a Group name in the top pane, and click deny for all the permissions in the lower pane.

Repeat for every user and group. There are likely just 3: “SYSTEM” “Administrators” and “Users”

Click “Apply”

Verify that all permissions are set to deny for each group before hitting OK.

(Note, the first time I did this, it appeared that Antimalware Doctordetected the attempted change when I hit “OK”, and undid my change.  But apparently “apply” uses some other  mechanism. )

Repeate this process for anything that looks like an executable (.exe) or executable extension (.DLL). Also you might want to remove permissions from all the other files as well. some of them are data required for the malware to detect when it’s process has been killed, and re-spawn itself. If it can not read the file (Because read permission has been removed) it has a hard time coming back after it’s processes have been terminated.

#2 rename the executable, and the directories they are in

your goal in this step is to prevent the malware from being able to start the next time windows starts.

In my case it was

C:\Users\Dad\AppData\Roaming\21E71E594E5225D3E6D0124C5B60BFE6\newsecureapp70700.exe

After changing permissions I renamed the executable file, every file in the directory. Do this ONLY if the malware is in a directory by itself, if it is in a system directory (somewhere under \windows, just try to rename the executable)

and most importantly

RENAME THE DIRECTORY.

(ONLY if the malware is in a directory all by itself)

If the malware is in a system directory you can try to move the executable to another directory (but that likely will fail)

You might not be able to rename the executable (Because it is running)

Why rename rather than just deleting it?
Well, smart malware authors will not only be able to detect when a malware processes is terminated, but also detect, and prevent deletion of the executable and it’s directory. This is typically done when the malware discovers it can’t read or write files in it’s home directory. If you delete those files, it knows someone is trying to shut it down.

If you just rename the directory it is in, all it’s files are still available to it. Until you reboot.

When you reboot, there is an entry in the system registry that will try to start the malware, and since the name of the directory has changed, the system will not be able to find the files, and it won’t be able to execute.

#3 Boot the system into “safe mode”

Your goal in this step is to boot into an OS mode that will allow you to remove the malware.

restart windows, repeatedly hit F8 before you see the windows logo (sometimes after you see the bios messages, sometimes just during that black time before the windows logo)

A menu will appear allowing you to select which mode to boot the OS.

Select “Safe mode”

I do not recommend “Safe mode with networking”, as it might give the malware access to the internet and re-seed itself on your  PC.

At this point you may be able to get rid of the malware simply by restoring the system to it’s last save point. This did not work for me. So I had to troll through the registry.

#4 search for the executables in the registry and document GUIDs and other keywords

First, standard disclaimer. The registry is where the OS and All programs store important configuration information. These steps describe a process akin to doing neuro surgery on your system. change or delete the wrong thing, and the system will never be the same.

open the registry editor (from a command prompt or the start menu’s search bar) by typing regedit.

I strongly suggest making a backup of the registry before proceeding. (with “computer” selected, select file->export and save the registry to a .reg file somewhere)

First thing, search for “Antimalware Doctor” in the registry.

This particular malware actually set registry entries in the system to make it look like it was legitimately installed. There was even an entry “Antimalware Doctor” in the installed programs control panel. DO NOT BE FOOLED. it is _very_ easy to add entries here, if you know how. you just need to know how to add some specific data in the system registry.

So, first thing I did was search for “Antimalware Doctor” and found a wealth of information.

Specificlly I found the following in the registry

[HKEY_USERS\S-1-5-21-1662259190-1271292992-2768786752-1000\Software\Antimalware Doctor Inc\Antimalware Doctor]
"datarl1"="KRoAGVdOQwQVExEwGhdaBhw="
"datarl2"="KRoAGVdOQwEIAQglBAZaBhw="
"datarlA"="KRoAGVdOQwQVExEwGhdaBhw="
"install_time"="8/25/2010 5:06:54 PM"
"database_version"="259"
"virus_signatures"="60817"
"affid"="7070010002"
"coid"="21E71E594E5225D3E6D0124C5B60BFE6"
"nsaftscann"="2"
"nsa"="5"
"nsaftscanunp"="2"
"updids"="ciod225,"
"LastScan"="8/26/2010 5:18:15 PM"
"nsstupn"="1"
"nsstupunp"="1"
"nssettlf"="1"
"nsstupsn"="1"
"nsstupsunp"="1"

Note: the “coid” this looks like a GUID (Genuinely Unique ID). also the number 7070010002″ is similar to the executable name. Also “updids” is “Ciod225″ these all look like they might be references to other entries in the registry.

You will eventually delete this entry, but ONLY after finding everything else.

At this point you might want to look under the registry entries that control what programs start when the OS starts.

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\RunOnce
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce

Once you remove the malware’s entries in above 4 run keys, theoretically it should start up when you reboot. But, likely it has other ways to start up if not started by the system (such as getting started through the browser proxy re-direct)

#5 search the registry for those GUIDs and other keywords, and remove them

Any program in windows depends on the registry. So a malware author is simply not going to put all it’s vital bits of information where it is easy to find.

at this point search the registry and find references to the executable, and to the other IDs. for Malware doctor I found several, and here they are.

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\App Management\ARPCache\Antimalware Doctor]

Under the registry entry:
[HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers]

I found these two entries:

"C:\\Users\\Dad\\AppData\\Roaming\\21E71E594E5225D3E6D0124C5B60BFE6\\newsecureapp70700.exe"="WIN95"

"C:\\Users\\Dad\\AppData\\Roaming\\21E71E594E5225D3E6D0124C5B60BFE6\\upd_debug.exe"="WIN95 DISABLETHEMES DISABLEDWM"

And, remember when I mentioned that it was easy to add entries to the list of software installed on your system? This is all that had to be done to get antimalware doctor to look like a legit program on your system. Technically all you need is the registry key, and the entry “DisplayName”, and that will create a bogus entry that will show up as an un-uninstallable software package.

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Uninstall\Antimalware Doctor]
"DisplayIcon"="C:\\Users\\Dad\\AppData\\Roaming\\21E71E594E5225D3E6D0124C5B60BFE6\\newsecureapp70700.exe,0"
"DisplayName"="Antimalware Doctor"
"UninstallString"="C:\\Users\\Dad\\AppData\\Roaming\\21E71E594E5225D3E6D0124C5B60BFE6\\newsecureapp70700.exe /uninstall"
"InstallLocation"="C:\\Users\\Dad\\AppData\\Roaming\\21E71E594E5225D3E6D0124C5B60BFE6\\"
"NoModify"=dword:00000001
"NoRepair"=dword:00000001

Note: if you try to actually run the uninstaller, all you get is an internet explorer window open to the “purchase product” page for “antimalware doctor” this registry entry tells us that the executable file newsecureapp70700.exe actually pops up internet explorer.

Also I found a bunch of bogus entries under one of the run keys

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run]
"wamcenxros.exe"="\"C:\\Users\\Dad\\AppData\\Local\\Temp\\wamcenxros.exe\""
"aomcnxerws.exe"="\"C:\\Users\\Dad\\AppData\\Local\\Temp\\aomcnxerws.exe\""
"haw389r7uifhdfigdhudf"="C:\\Users\\Dad\\AppData\\Local\\Temp\\zgwqduyts.exe"
"Rbifuvupoqoxevu"="rundll32.exe \"C:\\Users\\Dad\\AppData\\Local\\cprfech.dll\",Startup"
"newsecureapp70700.exe"="C:\\Users\\Dad\\AppData\\Roaming\\21E71E594E5225D3E6D0124C5B60BFE6\\newsecureapp70700.exe"
"hsfio38fiosfh398rfisjhkdsfd"="C:\\Users\\Dad\\AppData\\Local\\Temp\\services.exe"
"trawgd327uhf838jdfdsfdfds"="C:\\Users\\Dad\\AppData\\Local\\Temp\\drweb.exe"
"lhmerppl"="C:\\Users\\Dad\\AppData\\Local\\fflolomif\\vwmpqicshdw.exe
"Whabivanomozo"="rundll32.exe \"C:\\Users\\Dad\\AppData\\Local\\apogufutufumulig.dll\",Startup"

This almost made me think I had more than one creepy thing going on…

either way, you can see that modern malware is not simply one component, but a carefully crafted set of components. All designed with one purpose, prevent you from removing them, and directing you to a website for extortion of funds.

Also I found some bogus entries under the windows reg key:

[HKEY_CLASSES_ROOT\Local Settings\Software\Microsoft\Windows\Shell\MuiCache]

Entries under that key usually refer to programs that lie under the /windows or /program files directories. But I found:

"C:\\Users\\Dad\\AppData\\Roaming\\21E71E594E5225D3E6D0124C5B60BFE6\\newsecureapp70700.exe"="newsecureapp70700"
"C:\\Users\\Dad\\AppData\\Local\\fflolomif\\vwmpqicshdw.exe"="vwmpqicshdw"

#6 Finally, delete the registry entries for the executables, and the executables themselves.

Brief step. I took note of the OIDs and file names of the registry entries I found. Whenever I found a registry entry for the malware I saved that key off to a file.

You might thing this would be a never ending process, but it turns out there where only a few keywords used repeatedly

vwmpqicshdw
21E71E594E5225D3E6D0124C5B60BFE6
newsecureapp70700
Antimalware Doctor

I removed all registry entries with those keywords. Then finally went and removed the following Directories

C:\Users\Dad\AppData\Local\fflolomif\
C:\Users\Dad\AppData\Roaming\21E71E594E5225D3E6D0124C5B60BFE6\

#7 Fix the bogus “Proxy” settings

Malware Doctor did something extra Nasty. They set up a bogus proxy, so that every network request would be redirected through an IP address they could control. In this case, that IP address was your own PC. There are two ways to fix this, through the Internet settings control panel, or through the registry. Most people can figure it out through the control panel. But if you are comfortable with editing the registry. you just have to know which reg entry to tweak:

HKEY_CURRENT_CONFIG\Software\Microsoft\windows\CurrentVersion\Internet Settings

simply set the value of “proxy enable” to 0 (zero) (right click on the word “Proxy Enable” and select “modify” set the value to 0 and click OK.

Note: if you are reading this looking for electronic forensic information on this malware, you might find the following interesting, from the arp cache in the registry:

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\App Management\ARPCache\Antimalware Doctor]
"SlowInfoCache"=hex:28,02,00,00,01,00,00,00,00,a0,13,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00,00,43,00,3a,00,5c,00,55,00,73,00,65,00,72,00,73,00,\
 5c,00,44,00,61,00,64,00,5c,00,41,00,70,00,70,00,44,00,61,00,74,00,61,00,5c,\
 00,52,00,6f,00,61,00,6d,00,69,00,6e,00,67,00,5c,00,32,00,31,00,45,00,37,00,\
 31,00,45,00,35,00,39,00,34,00,45,00,35,00,32,00,32,00,35,00,44,00,33,00,45,\
 00,36,00,44,00,30,00,31,00,32,00,34,00,43,00,35,00,42,00,36,00,30,00,42,00,\
 46,00,45,00,36,00,5c,00,6e,00,65,00,77,00,73,00,65,00,63,00,75,00,72,00,65,\
 00,61,00,70,00,70,00,37,00,30,00,37,00,30,00,30,00,2e,00,65,00,78,00,65,00,\
 00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
 00,00,00,00,00,00,00,00
"Changed"=dword:00000000

I suspect there is lots of juicy information on the route this malware took to get to my PC. If I knew how to decode the arp cache.

In their second annual survey on “Virtual Goods Market Penetration and Growth in North America,” the two firms found that the median of digital goods purchases grew 67 percent from $30 in 2009 to $50 in the current year. More than one-fifth, or 21 percent, of digital goods buyers said they intend to purchase more in the next 12 months, too.

Not only are women a major market in games using virtual goods, the spend on virtual goods has increased.

Research firm Frank N. Magid Associates and in-game commerce platform PlaySpan report that 13 percent of the North American population (12 to 64 years) with internet access purchase digital goods, and that those consumers spent an average of $99 on those items in 2010 — a 14 percent increase over the average of $87 spent in the previous year.

This strikes me as a measurable progress toward a long time goal of the games industry: Make games women are interested in playing (and thus paying for). For decades game developers in NA and EU have struggled to figure out how to engage women. For years, some would say, whole console generations, where interesting to a majority of teenage males. My position has been that for the game industry to survive it not only needs to broaden market appeal to both sexes, but also a much broader age bracket.

So, Is this report a sign of Victory(is this as good as it is going to get)? or a signpost on the road to a bigger and broader market?

Click Me

Background

After a long, day job induced, hiatus from Unity 3D, I wanted to put my first unity project to bed before moving onto my next project (a facebook unity integration). After much clean up and fuss it stabilized and has bee posted to it’s own web page. I wanted the game to look clean, which meant I wanted the following:

• The player’s score should be displayed in the web page, not within the unity client
• Starting a new game should be done from buttons on the web page
• Controlling the “view” (rotation & zoom)

So this meant that I needed to post data from Unity to the web page, and call functions within unity from the web page.

Putting out

The first thing to do was figure out how to get player’s score to be displayed on the web page itself. This turned out to be fairly simple. You need 2 basic things

• A function defined in the web page for Unity to call
• use the “Application.ExternalCall” function from within your unity projects

Well, you also need to know how to change text in a web page, which it turns out to be pretty simple.

Changing IDentities

Some fantasy series I once read used a plot device whereby if you knew a things name you could control it. Well, in the web, if you give a thing a name, you can do all sorts of strange and wonderful things to it.

In the web page, giving something a name, is done by using the id tag. The official definition of ID is found here.

This may be best explined in an example. In my game page I have a “Red player score” and a “Blue player score”. In the web page I highlight each score by bolding it.

<b id="RPSScoreTxt">Blue Player Score</b>

Just a rule in the fantasy land that is HTML, if you know the name of something you can completely change it. Everything in an HTML page is an object, or more specifically, and html page is represented programmatically within the web browser as an object hierarchy, or Document Object Model (DOM for short).

When the page loads (remember this detail)  the text “Blue Player Score” will be displayed in it’s location within the page.

if you know an object’s name you can get a handle to that object within the DOM. Once you have the handle to an object you can  fiddle with the attributes and data attached to that object.

Will it come when I call it?

Having an element with a named ID allows you to get a reference to that object within scripts. Within javascript the object “document” represents the web page within the javascript is running. getElementByID is a function attached to the document object.

What is all powerful in a name?

The next trick is know what to change.

In HTML there is an attribute to every object that is nearly all powerful. The innerHTML attribute allows you to modify or replace the contents of an object. In this the inner HTML of is just some text. (But I did an experiment where unity replaced a simple text entry with an entire table).

function SetRPScore(arg){
 document.getElementById('RPSScoreTxt').innerHTML = ' ' + arg;
 }

So, if you make the call

SetRPScore("i did i bla bla");

The text “Blue Player Score” would be changed to ” i did i bla bla”

Remember the above function needs to be defined within <script> </script> block.

<script language="javascript1.1" type="text/javascript">

function SetRPScore(arg){
 document.getElementById('RPSScoreTxt').innerHTML = ' ' + arg;
 }
</script>

That was the hard part

now you have an object with a name you can use to get a handle to. You have a way to change that object, you have a function that takes an argument. The only thing you need to know is the Application.ExternalCall.  That function takes 2 parameters, the name of the function, and a parameter to pass to the function.

Application.ExternalCall("SetRPScore", 22 );

Bottom line (of code)

so if you want your unity game to make a change, such as set score, in the web page just do the following

• In your HTML page, give a chunk of text (or anything else) an ID
• Create a function(again in your html plage) that changes the innerHTML of that ID
• Call that function from within your unity plug-in via Application.ExternalCall

And that’s how it is done. To see the full details look at the source of my Aqueme game

Come back later and read my next segment, which go through the more terror prone activity of making a call into the unity plugin from the web page.

A Brief Word About Apple iPhone OS 4
As most folks already know, Apple introduced a new Terms of Service (ToS) with their iPhone OS 4 that has some Unity users concerned. While we don’t yet have a definitive statement to share at this time, we have a healthy, ongoing dialogue with Apple and it seems clear that it’s not their intention to block Unity and will continue our efforts to ensure compliance with the Terms. While it’s possible that ensuring compliance could mean additional engineering on our part, our committed goal is to provide you with uninterrupted ToS compliant tools and we think we’ll be able to do that.
If you want to stay up-to-date on our official response to this situation then please keep an eye on our company blog, and in particular posts from our CEO David Helgason: Unity Technologies Blog » David Helgason. We want to thank everyone out there for their patience while we work through these changing circumstances. We know it’s difficult to proceed with uncertainty, we’re doing all we can to ensure a positive outcome.

Maby you want to create a more productive work environment. Maby you are astounded by the volume of applications created for systems like facebook, the iPhone, Unity, Flash.

Maby you are just disgusted by the volume of flash games out there compared to the volume of apps that actually help people be more productive, accomplish goals.

Maby a new mental model will help.

Think of these systems as a riverbed, and the applications in them are like water.

and everything that has gone before is like Henry Ford’s factor pumping out Model Ts.

HUH?

Is this just meaningless new age drivel? This blog author hopes not.

How can there be 10s of thousands of apps for the iphone, facebook etc. etc. without having thousands of programmers in huge buildings managed by an army of MBAs? The average iPhone app looses money.  How can this be? Has everyone gone crazy? It is not sustainable!

Well, I believe there may be a “philosophical” description that sets Facebook, Flash, iPhone, apart from what has come before.

These new systems are more like a riverbed, which  allows water to, easily and  quickly, flow toward it’s own natural and desired destination.

Is a riverbed a thing?

What if we can not know, ahead of time, what people’s expectations will be? How do we make products and services that will exceed something we can not know before we begin product development?

Well, don’t make a thing, do not make a product. Make something that lets other people make products and services. This is not a new idea. Many names have been given to these type of things: Platforms, Enablers, Frameworks. Regardless of the name, they are systems for allowing the mass production of NON identical items in an environment DEVOID of requirements. “It must run on the iPhone” and purchased through the app store is NOT a Functional requirement.

Apple and Nike are not cool

They “enable” cool.

I know I’ll take a lot of flack, but I say even Apple and Nike do not make cool. It is the people, the community, especially the prosumers that make Apple, Nike, Burton, all the rest. Snowboard, skateboard, marketeers have know this for decades.  It is what prosumers do with produce that makes something cool. That is how Apple and Nike get to be seen as cool.

The tyranny of requirements

“You find a need, and you create a business to fill that need” the old saying goes. It is the basic, fundamental, notion behind traditional business development.  If someone has defined a need, that implies there has already been a situation that has been unresolved, or not satisfactorily resolved. “I needed to do X and I couldn’t” or “Y took me 7 hours which meant I could not do Z”. If we just look at needs we form requirements that make what we do faster and easier.

Pot scrubbers where invented to make cleaning faster. Dishwashers where invented by someone who answered the question “I expect someone to invent a machine to do it for me”. This is a well known, understood and discussed result, many texts on usability and design make this point.

Exceeding expectations would be inventing an eating method that does not require cleaning (Do paper plates qualify?)

It has been discovered that discovering underlying expectations, and exceeding them, leads to not only new products, but whole new industries.

Don’t touch that

The simplicity and rich feature sets provided by platforms such as Facebook, iPhone, Unity, increases productivity, and lowers the barrier to entry for all developers. Just like the PC had a much lower barrier of entry compared to Mainframes. This benifits internally developed apps(from the developer of the platform) as well as externally developed apps. The focus on simplicity also allows for high levels of automation. For apple, facebook, myspace, xbl, mixi, content is more like a river that flows through the streambed they have built.

I would compare it to a more traditional model that treats content more like cars through a factory. A factory produces a large number of individual items, all of the automation touches every part of every car made. The automation allows for a large number of cars to be made. Industrialization allows for a lower barrier, and higher volumes than building the products one by one. But it has it does have its limits.

In an industrial model, the creation of individual products is highly automated. This insures consistent product quality, and a relatively high rate of production. As long as each object is identical, and the requirements of each object is known ahead of time. Even with high levels of automation each individual component of each product is touched by the automation several times during the process. This inherently limits the potential output as well as limits the ability for external collaboration(Since the producing organization has to own and centrally house all the automation).

Don’t Require anything

As mentioned above, product produced through an industrial process have to be specified up front. Each component of a product must be completely understood so it can be manufactured. These specifications stem from Requirements. Every industrial process produces products or components “To customer specifications” or “conform to customer requirements”.

How many social iphone apps, do you think, where developed based on requirements? Did someone say “I require an application that lets me make shooty noises with my mobile phone”

Conversely how does the legacy industrial system deal with requirements like “I want it to be cool”?

So What?

So if you want to replicate what Apple,  Facebook, or other like companies have done, there are some things you need to do:

Realize you are enabling others to achieve their goals and desires. What you create is something that enables others to achieve their goals, not yours

It has to be simple, from every angle. Simple to use, simple to develop for. Don’t bind it up in corporate IP protection contractual obligations

It has to be compelling. People have to want to user your system to achieve their goals. if they don’t want to use your system, they will do it some other way, or not at all.

It has to be automated. Armies of people can not create and operate these systems, the model just breaks down. Every time someone involved with your platform has to touch something, like 3rd party content, it reduces your ability to scale, as well as limits everyone ability to contribute.

it has to be focused on the public. This does not really mean a public web site, but that the development and operational environments are focused on the end user/end developer, the public. Having individuals and organizations striving to meet internal goals are actually counter productive. (A great example is having an Entertainment companies  IT department report to the CFO rather than report to the producer or the head of product development)

Cafe’ world, Zynga’s fastest growing game. It is estimated to have
over 100 million unique active visitors per month. In ad market terms, that is 1/3 the US population. Compare that to the superbowl (nearly 100 million according to neilson).

One of the interesting facts that I took away from the virtual goods summit is the fact that zynga sees 10 dollar a year ARPU (Average revenue per active user over 1 year).

I know, according to all the VCs, that the advertising revenue model is dead, but seriously, placing an ad on these facebook games is dirt cheap compared to the superbowl. AND the quality of the data you can get from a social media campaign is tremendous.

It takes an abused metaphor to make something grow huge, fast

It takes a village to. We have all heard that cliche. Growth like what we see in the social media market needs support. Need creates demand, demand typically filled by meta industries that sorround and support the primes (Prime content developers, zynga, playdom, playfish). The social media meta industry is blossoming: Monitization infrastructure is now an industry (we’ve known that for a while, since we have been working with vindicia, aria, etc. etc.). Metrics companies that compete with cognos are sprouting, as we know there is a huge gap in the promise of cloud and cloud like infrastructure. For once, development tools are keeping pace, mostly because the state of the art is very simple apps, built mostly around flash, and the very very simple facebook API.

Marketing is not my thing, making stuff work IS

That simplicity does not yet extend to the back end technologies that run most social media networks. The LAMMP stack is not keeping up(Linux Apache MySQL MemCacheD PhP). It is a technology base, and architecture, that has not changed much in the last 15 years or so. It is  based on an architectural pattern that dates back to the late 70s(I am referring to MVC). There have been enhancements,  but no fundamental changes to the pattern.   load balancing, concurrency, and scale out are not natural to the pattern.

I would rather have pie

Facebook’s architecture is a step toward the ideal, squishing the technology stack down to only 3 basic layers,  but the architectural ideal would be to flatten the architecture down to one single layer. Rather than web sites being modeled after a layered cake, ideally they would be modeled after a pie (mmmm pie . A single amalgam of infrastructure, that can grow “out” as much as needed. insert mouth watering allegory to pie metaphor here.

Personally, I think we are close. One of the technical reasons for multi layered approach to web site design is the capabilities of each individual component(server). Another reason (or side benefit, take your pick) is security.

Why we have cake instead of Pie

The Multi layered architectural model is a result of economics: Expensive computers are optimized for various workloads. Basic economic theory, when a resource is scarce, it will be optimized for it’s largest most profitable market. This is the driving principle behind server design for the last 50 years. Computing power has been scarce, so the computers themselves have been optimized for specific workloads, like running databases. In the late 90s we saw the rise of generalized high density servers, and web server software, dominating the web tier, then a few years later, dominating the app tier. Now we are seeing these generalized high density servers becoming capable of running databases. So, finally, the same high density, low cost, servers are physically capable of running all the software systems used by high scale internet web sites. Which is a step. We are only a generation, or so, away from having a single physical server have enough capacity to run the entire stack of software for a high volume web site. For small websites, we are already there. Most word press based blogs use the same basic architectural pattern as a huge social media web site(web server, app server, database).

The management overhead for running the entire software stack is still too much for a single commodity server. Specifically VMWare and Zen combined with the OS still takes too much compute power away from the application. High volume web sites are still running their entire server farm at very high utilization rates.

Tipping point ahead?

Two things can/will happen.

1. The next couple CPU generations will begin to create the kind of compute power that web sites require in order to completely flatten the physical hardware stack.
2. The movement  of the virtualization functionality down into the hardware (arguably where it belongs). Ideally there should be code in the firmware of the chipset on the server motherboard that provides much of the low level functionality now provided by VMWare/Zen.

The Physical ideal

The ideal is this: A single, low cost, high density server with enough CPU cores, Memory and memory bandwidth to run OS instances for the web server, app server, and Database, as well as run in-memory IP networks and firewalls between the layers. AND provide lower latentcy and higher concurrency than individual systems connected by networks.

Once this physical ideal is reached, then we can develop enough experience to create a new software pattern that more naturally fits network application development.

So what, Now and Later?

So where to invest, where will contributions have the most impact, where are the control points?

In the short term, Anyone working on low level virtualization technologies. nVidia could make a short term killing by coming up with a chipset that could allow Zen/VMWare to virtualize a system with GPUs. (yes GPUs, a surprise technology in server side computing, when physics and stream based processing become important to increasing concurrency, and reducing latency). Also look to the industry veterans, Intel and AMD.

Long term investment, anyone doing research into a new architectural pattern that will replace MVC.  MVC has been in use for nearly 30 years now. The next architectural pattern will be developed in a fraction of the time, and because of the fundamental change in the economics of computing, it’s adoption rate will be unlike any seen previously.

It all comes narrowly back

This accelerated adoption cycle will dramatically narrow the number of players in the meta industry (and you thought I was just rambling, and that thread went nowhere). Put bluntly, when the architecture and techology replacing MVC comes along, it may be only come from one organization. With a likely adoption rate so accelerated that competition has no time to develop.

Interesting times ahead

Imagine 90% of the web traffic on the internet switching from one architecture to another in under a year. If zynga can develop and deploy an app in 5 weeks, that reaches 50 million people in 3 weeks. As the farmer used to tell me “that tells ya somethin right ther”.

How does Unity3D do XML?

Well, Unity3D uses the Mono implementation of the .net XML processor. Honestly I didn’t really want to know this, I just wanted to process XML. [did I mention that I am both needy AND demanding?].

In Unity3D (for 2.5 and 2.6 at least) it turns out that XML is best processed through C#.

What you need to do

• Write file open functions in C#
• Write XML read and write functions in C#
• write a function to read  the XML and process the results
• write a function to write game data in XML
• Optional: Read and write lists

The instructions are a bit confusing, you may just want to Skip them and, just read source code: Read and Write XML from Unity3D.

Calling C# from JavaScript in Unity3D

My project is coded in JavaScript, because it is the most productive. But, even though calling the .net functions can be done from JavaScript, it is still a bit of a mystery.  Processing XML is most natural through C#. So now I will have a project that uses both C# and JavaScript.

Fiddly bit of Unity3D magic

The first thing you have to know is that the C# files have to be “built” first, so the class names can be known to the JavaScript compiler.  The Unity3D documentation mentions that files in the Plugins or Editor folders get built before JavaScript.

1. Create a C# file, place it in the Plugins directory

I’ll cover how to call the C# stuff from JavaScript later.

Stuff you need to include for C# into Unity3D

using UnityEngine;
using System.Collections.Generic;
using System.Xml;
using System.Xml.Serialization;
using System.IO;
using System.Text;

Unfortunately this (currently) causes a cascade of includes, and you will notice the size of your plugin BLOAT bigtime.

Opening the file

It turns out that you will want to open your save file one way for read, and another way for write.

Most of the howtos on the web show reading and writing an XML file by opening a Stream. I found it is MUCH easier to open a stream to write, and open a “TextReader” for read.

but first, your C# class needs to be defined:

public class GameXML : MonoBehaviour
{
 string _FileLocation,_FileName;
 StreamWriter _writer;
 TextReader _reader;

Then you can open the file. I created one function for read and another for write.

public void OpenSaveFileForWrite()
 {
 FileInfo t = new FileInfo(_FileLocation+"\\"+ _FileName);
 if(!t.Exists)
 {
 _writer = t.CreateText();
 }
 else
 {
 t.Delete();
 _writer = t.CreateText();
 }
 }

 public void OpenSaveFileForRead()
 {
 FileInfo t = new FileInfo(_FileLocation+"\\"+ _FileName);
 if(t.Exists)
 {
 _reader = t.OpenText();
 }
 else
 {
 Debug.Log("SaveFile NotFound");
 }
 }

Yeah Ok, so I am not taking time to figure out indenting. You can check out formatted code in the included sourcecode.

Using Unity3D to write XML, it really is just two function calls

Well, you need a class that describes your games’ save data, but once you have that, all you need to do is turn it into a single text string (Serialization) then write the file out. Also once the data has been serialized, it can be written out using debug routines or printf or whatever, it’s just a big text string.

public void WriteGameDataAndTurnSequence(GameSaveGame SGDat )
 {
 string _data = SerializeGame(SGDat);
 Debug.Log("Game Sequence looks like:" + _data); 

 _writer.Write(_data);
 }

That’s it…Well sort of. The GameSaveGame class needs a little esplination. The function above is called from JavaScript. You can pass a reference to a class back and forth between JavaScript and C#. As long as that class has all the data you need then your good. BUT the tricky bit is building a class structure that describes all the data you need. For most games you’ll have structured data and lists that need to be read and written. This can get a little messy.

Templates, and sequences classes, Oh My

Your Serialize function will serialize ONE instance of a class. So in order to read and write complex data, you need to create a russian doll set of class constructs. Lists of class elements held by a “top level” class.

Ok, this gets messy. I want to write out a list of turns. the .net serialization can serialize a templated List. This is powerful, it means I can write out a list of arbitrary structure.What you need to know is that you will likely want to define 3 classes:

That One class can contain a Sequence(and tons and tons of other stuff). The Sequence is a class that contains a list of class instances. Your sequence is essentially defined as  a List<ListElementClass>. The Class definition the Sequence, actually sequences is the class definition that holds your actual game data.

Confused? I sure was.

Bottom line, create 3 class definitions in your C# file.

1. One “top level” class that defines the top level structure of your game
2. One (or more) “Sequence” classes that hold lists of data
3. One “List Element” class that describes the data in each list element

Top Level

So my “top level” class just needs to really define two members, one for “settings” and one for the game’s turn list.

public class GameSaveGame
{
 public GameData GameSettings;
 public GameSequence Turns;

 public GameSaveGame()
 {
 GameSettings = new GameData();
 Turns = new GameGameSequence();
 }

 public GameSaveGame(GameData pData, GameSequence pGS)
 {
 GameSettings = pData;
 Turns = pGS;
 }
}

Sequence

my Sequence  GameSequence class so that it would be easy to add elements and stuff like that.

public class GameSequence
{
 public List<TurnData> TurnList = new List<TurnData>();
 private int Sequence = 0; 

 public GameSequence () { }

Thee are more convenience functions in the source code, but you get the Idea. Finally, the sequence is a sequence of “TurnData” structures. The “TurnData” class actually describes what a turn is.

List Element

Finally, the inner most nugget. A simple class that describes the data in each turn.

public class TurnData
{
 public TurnData (){ Unconditional = false; SequenceNum = 0;}
 public TurnData ( int x, int y, int z, int PID, bool cond, int seq)
 {
 X = x;
 Y = y;
 Z = z;
 PlayerID = PID;
 Unconditional = cond;
 SequenceNum = seq;
 }

Reading XML with Unity3D

Reading is just as easy as writing, but I recommend using a different type of stream to read and parse the XML. Extracting XML into game data is just two steps

1. Open the file for read
2. De-serialize

Open the file for read….

public void OpenSaveFileForRead()
 {
 FileInfo t = new FileInfo(_FileLocation+"\\"+ _FileName);
 if(t.Exists)
 {
 _reader = t.OpenText();
 }
 else
 {
 Debug.Log("SaveFile NotFound");
 }
 }

Reading XML with Unity3D: a little magic

Here we actually “do something with XML”. XmlSerializer is the function that parses the formatted xml data and fills in all those classes you defined.

GameSaveGame DeserializeGame(TextReader file)
 {
 XmlSerializer xdsg = new XmlSerializer(typeof(GameSaveGame));
 GameSaveGame newGame;
 newGame = (GameSaveGame)xdsg.Deserialize(_reader);
 _reader.Close();
 return newGame;
 }

within Unity3D Calling C# from JavaScript

So, one last thing. How to call the C# functions from Javascript.

1. Declare a variable using the name of the C# file (becomes the class name)
2. Get the instance of that class
3. Fill a data structure with your game save data
4. call your unity3D save/load function

It looks something like this

private var SaveLogic : RnWXML; // note I put my CSharp files in the Plugins directory
SaveLogic = GameRootClass.GetComponent(RnWXML);
var NewGame : SaveGame = SaveLogic.ReadSaveGame(); // the SaveGame class is defined in SaveLogic C#

I hope this help someone. There are a lot of twiddly bits of magic that need to be strung together in order to read and write XML. What I have NOT done yet is read and write XML from a URL, which will mean opening up a stream reader and writer to an HTTP connection.

That will be interesting, facebook might be a good source of XML….

And if you missed the source code link at the beginning of this post: Read and Write XML from Unity

Where to Developers get their info?

You can take official courses in universities, and from organizations like Full Sail and Digipen (and LA Film School, and Art Institute, and check your local university).  Or you can get a traditional degree in computer science, graphic design, and make sure to pick up classes on game development and interactive design.This is a great way to go if you are nearing, or just finished high school.

What if you already way beyond that?

What if you have heard how easy tools like Flash, Unity3D and Torque are to use, and you want to know more.

What if you are just curious?

What if you just need to do some research, but don’t really need a working knowledge?

Well, guess what, you have arrived at a time just when a golden era of community supported development is on the rise. or if you like sound bytes

Let google be your guide, teacher and mentor

In other words “If you want to do something with a technology, someone somewhere has likely done it already and  have bogged about it, and likely shot a video tutorial posted on youtube and vimo”.

Instructing you : opposite directions

Over the past 2 years there have been two completely divergent trends: 1 away from centralized sites with specific topics, and 2, seemly contrary, the emergence of generalized sites with tons of information on how to do everything.

For developers, or the developer curious, or the would-be developer, improvements in google allow anyone to  find whatever they are looking for, no mater wherever it is. Why is this earth shattering? well, in the stone age, before google (like 1998), if you wanted to know how to do something, you likely had to physcially travel to a centralized repository of information, and manually search through physical materials(it was called a LIBRARY). Interesting thing, there is a branch of science, and at least 2 industries and business models that exist to support that method of finding out how to do something.

But now it is the 21st century, and thank the internet we don’t have to do THAT anymore. As liberating as libraries where, free internet search engines are a tru quantum leap.  Now most info comes from individual sites, found through specific google searches. If we want to know something, we google it. If an individual wants to know ANYTHING, all they have to do is google it. (Sidebar, there are some interesting behavior shifts going on. Now you do not need to understand, or know how to do something before embarking on that thing, you can just google as you go. Think of it as Just-in-time-Know-How).

Back in the library days, information had to be researched, gathered, co-lated, and organized in a manner that made it easy to search by humans. Developers of technology spent large portions of their budget on documentation and training. Now with internet search technology, as soon as someone learns how to do something they can post it on their blog, or create a youtube video. Anyone anywhere on the internet can find that tasty tidbit of information. (assuming the author uses good keywords, and the searcher does to)

It takes a community to raise a game developer

How does this notion applyto game development? Most organizations that make technology for game development now-a-days rely on “their community” for a large poart of information, education and training materials. For a given technology there will be a main wiki, a primary forum, but then also dozens to thousands of individual blogs and websites. Excellent examples are the communities that support the Unity 3D , and Flash (action script) technologies. The best way to find out anything about these technologies is to start with google. Of course there are some basic tutorials provided by the purveyors of each respective technology, but the tutorials found on the web are specific to each individual goal.

Is it all coming back together?

The flip side of this is the emergence of websites like ehow, howstuffworks.com, soyouwanna.com, etc.. A few years ago people started to notice that the ad revenue for youtube videos that provided tutorials was very different than the rest of the whole site. Ad sellers, and buyers, learned that you where much more likely to sell something to someone asking the question “how do I”. This has coincided with a re-emergence of the  good old yankee inclination to tinker, referred to in it’s current incarnation as “the Maker Movement”.

Both of these methods of learneing are supported by a growing internet culture of “learn & teach”. When people learn something, figure out how to do something, they blog it, tweet it, shoot videos of it.. These blogs, tweets and videos feed both the individuals, and the big web sites like ehow.

Another emerging trend: professional Micro “how to videos”:

There is a small community of videographers and writers, mostly in and around Hollywood, that shoot short video tutorials. “Microproductions” I think they are called. The deal is, when a website like ehow encounters a prospectively profitable set of “how to” queries, they will contract a videographer to go film a bunch of very short how to videos. A videographer that specialises in this, is often a TV or Cinema camera opertor between jobs. They will get hired to shoot several dozen how to videos each day. Each individual video might only net 20 bucks or so from the site (like ehow). The production quality does not have to be high, and they can be done in masse. So a single day’s shooting can net a few hundred bucks, if the videographer is efficient and can direct the subject well.

Also some technical writers are turning to this business model. LearnMeSilly.com is a successful example of how technical instruction may be done from now on. One of that web site’s principals created some,  tutorials for Unity 3D. Now the author is publishing  a book.

The book will get published and find its way into Libraries.

I coined this phrase a year or so ago “Welcome to Early 21st century pop culture, now with video tutorials!”. (reference to www.knowyourmeme.com).

• Earned

• Purchased

Earned currency is generally given to participants in your virtual economy for work done or time spent. Purchased currency is just that, purchased with cash outside your virtual economy. (kinda like if you dig up gold and exchange that for money, in the real world).

I am purporting there is a 3rd type of currency that can be applied to virtual economies, Achievement Currency, a type of reward for doing accomplishing a goal in the virtual economy. This currency is tied to specifc, pre-defined, and known goals. Complete the quest for the golden sword, buy 1000 coins, posting 10 answers in the forum (that receive positive feedback rating), gifting 100 items.

Achievements are now a well understood reward/motivation game mechanic. But using the same mechanic as a currency in virtual economies is a new concept.

Earning achievement currency is relatively easy concept. Think of “badges” or “Medals”. But how do you spend them? It is hard to unlock the mind from the concept that an award for achievement can some how be “undone” or “given away”. It also contrary to think that achievements could be purchased with cash.

Achievement currency, when added as a 3rd type of currency in a virtual world certainly have the ability to deepen the experience in a virtual world, but if it is too easy to buy them with cash there will certainly be members of your community to vocally decry “posers”.