[Nibbits Tools] SC2Replay Parser

Page 1

As many have noticed, we are parsing SC2 replays here automatically on Nibbits. This is done via some tools we have written in Python, that tie directly into the site (being that it's built on Python). Being that I'm constantly asked how some of our tools work, we are beginning to open source some of the components.

First off, let me start by saying all components are open sourced under the original BSD license (meaning you must give credit if you use our material). They can be obtained via at our repository on GitHub.

The component I'm going to be talking about today is our SC2Replay parser. It's not 100% complete as we're working out a few of the last kinks, but it includes what is assumed to be the entire structure of the SC2Replay/replay.info file.

Usage is fairly straightforward:
```
Parses out SC2Replay files.

>>> from common.parsers.replays import ReplayInfo
>>> from common.bitstream import BitStream
>>> 
>>> fp = BitStream(open('Path.SC2Replay', 'rb'))
>>> parser = ReplayInfo()
>>> data = parser.serialize(fp)
>>> print "Map filename: ", data.GameInfo.MapInfo.CachePath
```
You can also test w/ this file directly:
```
python replays.py 
```
Have feedback about Nibbits.com? Let me know!

Send Message Reply Permalink

Posted 2010-03-03
dcramer 17

Addict

Battle.net

Name:

zeeg.227

Realm:

North America
Awesome! Could someone create a C#/VB .NET version? :D

Send Message Reply Permalink

Posted 2010-03-03

AzureGlow 1

Registered User
AzureGlow: you could use IronPython.

Send Message Reply Permalink

Posted 2010-03-04

lastkarrde 5

Lieutenant
A PHP port would be hot.

Reply Permalink

Posted 2010-03-04

com

Anonymous
71.63.43.xxx
I didn't want to just consume it from .NET I wanted to see what a C#/VB .NET equivalent would look like :)

Reply Permalink

Posted 2010-03-04

AzureGlow

Anonymous
209.151.133.xxx
Hey, this is NewbiZ from Darkblizz.

I've been working on my parser since some time now, and I'm able to parse almost anything.
Just wanted to know if you would be interested to test the parser against Nibbits' database.
There are python bindings.

Let me know,
NewbiZ

Send Message Reply Permalink

Posted 2010-03-08

NewbiZ 2

Private

Battle.net

Name:

NoobiZ.noobiz

Realm:

EU Beta
Sure I can do that. Just throw me the source.

Have feedback about Nibbits.com? Let me know!

Send Message Reply Permalink

Posted 2010-03-08

dcramer 17

Addict

Battle.net

Name:

zeeg.227

Realm:

North America
What do you mean by anything? Do you mean to say you've cracked files other than replay.info?

Reply Permalink

Posted 2010-03-08

anon

Anonymous
71.63.43.xxx
Do you have any idea how to encode

replay.info
replay.game.events
replay.smartcam.events
replay.sync.events

to a regular *.SC2Replay file? I tried to encode these files with LZMA and changed the ending to *.SC2Replay but this didn't work. I also couldn't decode the regular *.SC2Replay files although they are supposed to be encoded with LZMA. I assume the coders from blizzard changed the header of the replay file in order to prevent it from being decoded. I also think that the header includes some information like map, player names, game name...

I can't figure out how the guys from blizzard are doing it. If I can figure this out it would be possible to create a streaming application like WTV for Warcraft 3.

Hope you have a small hint and can point me to the right direction. I really admire your work, keep it up!

Rgds
Layne

Reply Permalink

Posted 2010-03-09

Layne

Anonymous
84.178.41.xxx
Hey,

The project itself is hosted on my RedMine, available at:
http://projects.coderbasement.com/projects/show/sc2replay

And the public SVN is accessible at :
http://svn2.assembla.com/svn/sc2replay

You will need:
* CMake >= 2.6
* libmpq
* boost header
* boost.python (if you want to build the python bindings)

Instructions:
svn co http://svn2.assembla.com/svn/sc2replay
mkdir sc2replay/trunk/build
cd sc2replay/trunk/build
cmake ..
make

Use:
cmake .. -DBUILD_PYTHON=OFF
if you don't want python bindings, or don't have boost.python

You can try out tests/read_mpq, it'll display infos about the replay (players, map, races, colors, chat messages...).

---

For now, it is able to parse anything that is known in the replay.info file.
You can also retrieve chat messages from replay.message.events, but it'll fail if a player used the blink map function (it'll work soon).
As for the other files (game events & so on...) I plan to support them soon, they are reversed but not implemented ATM. Btw they arent really useful.

I've found that the parser is able to handle some replays that nibbits' one can't.

I'm interested if you can try it out against all possible replays and report any replay that's not parsed correctly.

If you have any requests, it's also welcome. Right now I'm trying to figure out how to get the winner(s) of the game.

---

@Layne: I did not understood quite well your questions.
MPQ files aren't just lzma encoded files. They are encoded in sectors with possibly different algorithm (even if lzma seems to be the only one used in SC2).

I suggest you to get an MPQ library that will handle this part for you (such as libmpq or stormlib). Reversing the replay format is hard enough not to have to decode MPQ manually ;)

Send Message Reply Permalink

Posted 2010-03-09

NewbiZ 2

Private

Battle.net

Name:

NoobiZ.noobiz

Realm:

EU Beta
dcramerAs many have noticed, we are parsing SC2 replays here automatically on Nibbits. This is done via some tools we have written in Python, that tie directly into the site (being that it's built on Python). Being that I'm constantly asked how some of our tools work, we are beginning to open source some of the components.

First off, let me start by saying all components are open sourced under the original BSD license (meaning you must give credit if you use our material). They can be obtained via at our repository on GitHub.

The component I'm going to be talking about today is our SC2Replay parser. It's not 100% complete as we're working out a few of the last kinks, but it includes what is assumed to be the entire structure of the SC2Replay/replay.info file.

Usage is fairly straightforward:
```
Parses out SC2Replay files.

>>> from common.parsers.replays import ReplayInfo
>>> from common.bitstream import BitStream
>>> 
>>> fp = BitStream(open('Path.SC2Replay', 'rb'))
>>> parser = ReplayInfo()
>>> data = parser.serialize(fp)
>>> print "Map filename: ", data.GameInfo.MapInfo.CachePath
```
You can also test w/ this file directly:
```
python replays.py 
```
Could you please write a few lines on how to set this up in ubuntu, for instance?

Cheers!

Send Message Reply Permalink

Posted 2010-03-11
poofus 1

Registered User
Any chances of someone porting this to PHP via an extension?

Reply Permalink

Posted 2010-03-16

Kris

Anonymous
76.93.175.xxx
Project moved to http://code.google.com/p/starcraft2replay

Kris> you could create your own wrapper using SWIG. It would not take more than an hour.

Send Message Reply Permalink

Posted 2010-03-25

NewbiZ 2

Private

Battle.net

Name:

NoobiZ.noobiz

Realm:

EU Beta
NewbiZProject moved to http://code.google.com/p/starcraft2replay

Kris> you could create your own wrapper using SWIG. It would not take more than an hour.

Did you get a chance to try out the bitstreaming yet NewbiZ?

Have feedback about Nibbits.com? Let me know!

Send Message Reply Permalink

Posted 2010-03-25

dcramer 17

Addict

Battle.net

Name:

zeeg.227

Realm:

North America
I tried indeed, but wasn't able to get the part between the "cache path" and the "needed files" to work properly.

That's not _that_ important since I can get all known information using byte parsing, but bitstreaming is definitely the way i would like to head to.

Send Message Reply Permalink

Posted 2010-03-25

NewbiZ 2

Private

Battle.net

Name:

NoobiZ.noobiz

Realm:

EU Beta
NewbiZI tried indeed, but wasn't able to get the part between the "cache path" and the "needed files" to work properly.

That's not _that_ important since I can get all known information using byte parsing, but bitstreaming is definitely the way i would like to head to.

Ya that's definitely the way to do it.

I had a chance to dig through your source, and strangely enough you're doing something very similar to what we're doing on Nibbits right now (the live code, not the public code).

Instead we just seek to the end, and reverse seek til we hit a specific number of 0x00 bytes.

Have feedback about Nibbits.com? Let me know!

Send Message Reply Permalink

Posted 2010-03-25

dcramer 17

Addict

Battle.net

Name:

zeeg.227

Realm:

North America
Hello zeeg and NewbiZ!

We are a small team working on a PHP library for parsing the replays. I've been in touch with zeeg previously but I just wanted to ask how you are doing? Has there been any development regarding the understanding of the replays?
We will happily provide any kind of help needed.

Our project: http://phpmpq.codeplex.com/

Send Message Reply Permalink

Posted 2010-03-25

ZetaTwo 1

Registered User
Hey ZetaTwo!

I wrote a post on your project forum this morning.
I think the best solution would definitely be to share our work in a single library, and generate bindings for PHP/Python/whatever, and not to create tons of parsers clones in all languages. I think Zeeg agree on that too.

Creating a wrapper can be done really quickly using SWIG. You basically just have to register on googlecode and start committing ^^

Send Message Reply Permalink

Posted 2010-03-25

NewbiZ 2

Private

Battle.net

Name:

NoobiZ.noobiz

Realm:

EU Beta
NewbiZHey ZetaTwo!

I wrote a post on your project forum this morning.
I think the best solution would definitely be to share our work in a single library, and generate bindings for PHP/Python/whatever, and not to create tons of parsers clones in all languages. I think Zeeg agree on that too.

Creating a wrapper can be done really quickly using SWIG. You basically just have to register on googlecode and start committing ^^

I hate Google Code btw ;)

Github 4 life!

Have feedback about Nibbits.com? Let me know!

Send Message Reply Permalink

Posted 2010-03-25

dcramer 17

Addict

Battle.net

Name:

zeeg.227

Realm:

North America
We're also working on a replay parsing engine at http://screplays.com
It has been functional for quite some time now but we are working on more advanced features.

I have created a thread regarding this on our forum:
http://screplays.com/forum/showthread.php?p=138#post138

We should team up on this project :)
Please contact me if you're interested in working with us. I will come and hang out in your IRC channel as well :)

Send Message Reply Permalink

Posted 2010-03-26

screplays 1

Registered User
I'm extremely newbie when it comes to Python and getting it to work with PHP, can anyone write a short tutorial on how to parse a replay with Python through PHP? I know all the ins and outs of PHP, but have no clue where to start with Python.

Reply Permalink

Posted 2010-04-12

Vash

Anonymous
134.71.145.xxx
tory burch sale
gucci borse
yeezy shoes
birkenstock outlet
canada goose
ysl outlet store
uggs on sale
longchamp handbags
gucci shoes
ugg sale
michael kors outlet online
mcm outlet online
michael kors handbags clearance
michael kors handbags outlet
christian louboutin uk
longchamp bags
lacoste outlet
coach outlet online
michael kors outlet canada
discount oakley sunglasses
jordan femmes pas cher
hejiexia201621111

Reply Permalink

Posted 2016-11-11

20161111 hejiexia

Anonymous
108.62.62.xxx
<P>Those adidas superstar who cheap jordans online are red sole shoes hesitant to jordans for cheap are moncler women charged christian louboutin sale actual mont blanc pens discount funding simply nike store by the moncler coats balmy cheap nike basketball shoes fluffy uncanny idea they get beats by dre studio promote ones nike outlet youth tend red bottom shoes for women to nike clearance be beats headphones on sale asked to nike outlet store donate under armour store if beats by dre on sale he or she new jordans go nike factory outlet physical cheap jordans online in retro jordans trade. nike factory store You and cheap jordans online even/or nike sale the christian louboutin sneakers kid retro jordans for sale earns something nike factory store launch cheap nike basketball shoes anything else nike outlet online up including xmas crackers cheap nike running shoes

Send Message Reply Permalink

Posted 2016-12-15

loemrntdherid1 2

Private
MYY

barbour jackets dsquared2 outlet mcm backpack outlet prada shoes dsquared2 suns jersey oakley sunglasses polo ralph jordan nike roshe michael kors outlet online sale nike huarache chiefs nfl jersey nhl jerseys christian louboutin converse shoes the north face outlet kate spade outlet nets jerseys thomas sabo timberwolves jersey boston celtics jerseys jordan release dates north face jackets beats headphones bears nfl jersey wizards jersey beats by dr dre browns nfl jersey woolrich outlet cheap jordans handbags outlet swarovski jewelry longchamp black friday oakley black friday puma giuseppe zanotti ray ban baseball bats rolex montre womens clothing pandora polo outlet fendi outlet cheap nhl jerseys lunette oakley steelers nfl jersey ralph lauren hollister p90x workout burberry outlet longchamp outlet cheap nfl jerseys cheap ray ban coach outlet christian louboutin outlet nba jersey ralph lauren outlet rams nfl jersey orlando magic jersey broncos nfl jersey iphone case christian louboutin shoes nike shoes outlet cheap jerseys hilfiger retro jordans nike air max saints nfl jersey adidas tn pas cher hollister clothing store new balance outlet ed hardy jets nfl jersey swarovski nike mercurial dolphins nfl jersey rolex watches for sale polo ralph lauren cheap jerseys true religion coach purses outlet eyeglass frames true religion jeans buccaneers nfl jersey ray ban michael kors bags occhiali ray ban burberry handbags michael kors polo ralph lauren soccer shoes outlet burberry outlet ralph lauren factory store nike free michael kors burberry oakley sunglasses outlet vans shoes cheap oakley sunglasses bcbg dresses warriors jersey mcm handbags replica watches rolex watches miami heat jerseys ray ban black friday burberry outlet online huarache timberland boots panthers nfl jersey coach outlet toms shoes grizzlies jersey oakley vault nba shoes celine black friday longchamp outlet nike store polo ralph lauren outlet relojes asics outlet nike free run ray ban mcm handbags michael kors tommy hilfiger air max salomon tory burch outlet charlotte hornets jersey air max mbt shoes barbour trail blazers jersey air huarache cheap nike shoes christian louboutin shoes kings jersey new balance katespade [url=http://coach.euro-

Reply Permalink

Posted 2017-04-06

myy

Anonymous
139.162.121.xxx
adidas outlet online
coach outlet
michael kors outlet online
canada goose jackets
lacoste clothing
uggs outlet
uggs outlet
ugg outlet
michael kors outlet online
ralph lauren
ray ban wayfarer
ugg outlet online clearance
tory burch outlet
uggs outlet
ugg boots clearance
ugg outlet
canada goose outlet online
canada goose outlet
fred perry polo shirts
canada goose jackets sale
ralph lauren shirts
ugg boots outlet
coach factory outlet
michael kors outlet
canada goose outlet
canada goose outlet
kate spade handbags
canada goose
michael kors outlet online
pandora jewelry
ugg canada
coach canada
true religion jeans sale
oakley sunglasses
mlb jerseys
ugg outlet
canada goose
nike outlet
prada outlet store
adidas yeezy shoes
air max shoes
uggs outlet
ugg outlet
coach outlet canada
moncler outlet online
pandora charms
coach outlet
mont blanc pens
michael kors outlet
polo ralph lauren outlet
polo ralph lauren
yaoxuemei20171208

Send Message Reply Permalink

Posted 2017-12-08

yaoxuemei 5

Lieutenant

Page 1

New Reply

If you were logged in you would gain 3 XP for posting a reply.

Support Nibbits by linking to us: