Developer console scraping, part two

I wrote a post about scraping Google’s Developer Console a year ago. A site visitor told me about his problems with this solution so I decided to help him out with a different option – a simple python script. It requires less prerequesites to be installed – only mechanize (sudo easy_install mechanize or sudo pip install mechanize).

In case you did not notice, when you log in to the Developer console, you can click on the Reports option and then statistics. There you have to write the name of your application and you will be presenteed with the statistics in zip files. I’m only interested in number of downloads and how many people actually use my application. Right click on Latest monthy install report and copy its url, we will be needing it shortly. For example, in my case the url would look something similar to https://play.google.com/apps/publish/bulkreports/download?period=2015_07&report_type=STATS_INSTALLS&package=com.example.application&dev_acc=0358642832969076833.

As you can see from the url, it contains a time period. The zip file also contains several csv files, that display daily statistics. I’ve made a simple script that downloads current months statistics in a zip file, extracts it and outputs the very last entry. That way I get the number of downloads and how many people have my app installed on their device. The script is available over here.

Before running it, you will need to do a few things first though. As stated before, install mechanize first. Get a hold of your dev_acc – it should be in the url you copied before. Take a look at the line that starts with result=br.open (should be number 22). Replace my dev_acc id with yours. Also replace the example package name with your application package name. Definitively replace the email and password with an account that has access to the developer console. I’m using /home/peter/production/scrape/ folder to hold  the downloaded zip and csv files. Change that according to your needs.

Now you can give it a go! The script first opens the url to the zip file that contains all of the statistics for the current month. It cannot, because it’s not authorised – it then logs in to the Developer console. In a moment the zip file is saved to the path you want. The file that ends with overview.csv is opened, because it’s the only one that contains the neeeded statistics. With the help of tail and cut, the needed information is parsed out. Then you can do what you want with it.

Developer console scraping

Google Play has a nice Developer console, which displays various information ranging from download stats to crash reports. It provides a great insight how the app is used. I wanted to automate getting the most important information from the apps I have published. I’m mostly interested in download stats and ratings. Tried several tools and even tried to make a simple python script. Because the site uses lots of javascript, there was no luck in creating my own solution. After some googling, I did stumble upon Google Dev scraper. It has been running on my Raspberry Pi for almost 4 months and it is truly working great!

google_play_store_logo

It needs some prerequisites such as Ruby and RubyGems. After these two are installed, clone the latest version from github. Because I want to get these information on a daily basis I’m using a cronjob that runs a bash script. Upon completion it creates a php file that displays the results in a simple manner. To start using the scraper, you need to get your developer Id. Visit the console and copy the numers after “dev_acc=”. Afterwards you need to get the package name for the stats that you want to get. You need to limit the search for the stats, get the publish date and latest displayed date in the statistics displayed in the console. When you have all of this information, issue the following command googleplay_dev_scraper -u [email protected] -p yourPassword -a yourDeveloperID app.package.name 20130227 20140106 >results.zip (replace the last two numbers with the starting and ending date in the same format!).

If you replaced all the data with your own login information, package name and correct dates, you should end up with a zip file. It contains lots and lots of statistics in csv format. Now it’s all upon you to decide which information you need and automate it with a bash script.  In general, my script does the following. Everyday at noon the starting command is issued to get statistics in a zip file. This file is then unpacked and removed.  Then the app.package.name_overall_installs.csv is read for the latest download number and ratings. Afterwards you can simply email them or display them in your website. Truly an amazing tool!

Android device as a security camera

I have a really old android phone sitting in my drawer. Its high time my Galaxy Spica got a new job and so can your old device.

Start with the easy part. Install SSH Server on your device, its totally free. This app will help you remotely reboot your device, so you do not have to touch the device, after you set it up. Make sure you enable auto start of the app on boot. When creating a new server for SSH make sure you enable auto boot as well and memorize the connection port. On users tab, add a user with the password. Try to connect to the device with putty if on Windows. Type in the IP of the device (set a static IP), port and user name. Now you should be able to issue commands such as restart for restarting or restart -p for powering off. In order to do this, you have to be superuser, so type in su first.android security camera

 The important app for streaming video is IP Webcam, free again. Just like SSH server, make sure you enable the setting that runs this app on boot. You can play around with other settings to fit your needs (such as lowering FPS, which helps with your device not overheating). Otherwise this app is ready to go from the start. Check out the stream by visiting 192.168.X.XXX:8080 (IP of the device along with :8080). Click on Flash tabs so you can see the video and audio stream. Really nice! Now you need to read this stream and get email notifications when a change is detected.

Last part is a bit longer, because you need to find the right settings for your camera, so you do not get too many false notifications. Install a nice application named Motion with apt-get install motion. You need to modify a few things in the configuration file with nano /etc/motion/motion.conf. Change daemon off to daemon on, which will start motion on startup. Now search for netcam_url and add  http://192.168.X.XXX:8080/video. This setting tells motion where the video source is.  The last of the mandatory settings is on_picture_save, which sets the action that occurs after motion is detected.  Change it to something like on_picture_save echo ‘Webcam alert’ | mail -a %f -s “Webcam Alert” [email protected]  Save the file, move your hand in front of the phone and hope for the best. You should get an email that has a picture attached. These pictures are saved locally, you can change the path by altering the setting target_dir.

This should be all there is to it. You can now fine tune the settings – which parts of the screen can trigger the motion detection, how many changed pixels trigger motion detection, video recording and much much more. Have fun!

Galaxy Ace S5830 flashing

There are loads and loads of tutorials for flashing custom ROMs to S5830 Galaxy Ace smartphone. I know, because I’ve read several of them. But there are a few main reasons for this post that has content, which already exists on lots of pages. I’m always a little bit afraid of flashing zip files from unknown sources. I never know if the ROM will be any good. I’m always afraid of botching up the flashing project. Lots of files needed for flashing are no present on hosting sites. These three reasons convinced me to write a tutorial that will contain 100% working files, the 100% correct flashing process and a really good custom ROM. All the files will remain on my Dropbox as long as I have one. This whole process has been tested myself and I can guarantee that you will not brick your phone if you follow this tutorial step by step.

If you never connected your Ace to the PC, you need to install Samsung Kies. It’s needed because it has various drivers bundled in its package. I had to install Visual C++ fix because I was getting an error concerning an dll error. Download ClockWorkMod 5, latest official firmware and Odin. Optionally you can also download root updater and Myth ROM. You will read about those two later on. Place all of these files, except the Odin package, on your SD card but do not insert it just yet.

It’s useful if your phone is charged over 50% just in case before beginning. If you press power button, volume down and home you will enter downloading mode. Fire up Odin,select OPS and find your ops file. Tick the One package option and connect your phone. Take a look at the screenshot to see how it looks when the process is finished. Double check everything and press start. Your phone will be rebooted several times, but do not worry.tutorial

Congrats, you have the most recent official firmware for this phone. If you want reliability and stability, stop here. But if you want some extra space and other cool features, then install a custom ROM. I recommend Myth ROM mod. The rest of the steps are not hard to follow so do not be afraid to keep reading. Power off phone, press power and hold it with home button. When the screen appears, stop holding the power button but keep pressing home button. Welcome to stock recovery mode. First you will install ClockWork Mod 5 that will replace the current recovery that you see. Select apply update from sdcard. Select upd_1 which will give you root access. Repeat the process but this time choose  Clockworkmod package. Reboot and return to the new and improved recovery mode. Navigate to Advanced, Partition, select at least 1024 (if your SD card has such space) and 32 for swap space. Now you can install the ROM. Run it the same way you ran upd_1. You will be presented with a really nice installation process. Select such options as you see fit. Select reboot and wait for the system to boot. It will take a several minutes for first boot but it is worth the wait! That is all, enjoy your your Galaxy Ace on steroids.

Easy Facebook hash generating

According to Google Analytics one of my previous posts regarding Facebook key hash generating is quite popular. I have recently stumbled upon an even easier method of generating this key. No extra code is needed at all, but it is limited to Windows users only. You need to download the application here. Afterwards find the keystore for your application, make sure you’re selecting the right one. Run the downloaded application, drag and drop the keystore to the application. Type in the keystore password and wait for magic to happen. I have checked the file with NOD32 and it’s clean, so no worries on that part. Easy as pie!

This great little application was not created by me, I’m just promoting it.

4LQX3

Simple Twitter integration for Android

I have decided to try out integrating Twitter into an Android application. It appears that this takes a little bit of more effort than for Facebook. There is no official SDK for this, so you are pretty much on your own. There are lots of unofficial plugins like Twitter4J but I could not find one that would be simple to implement and would look nice. In the end, after hours of Googling and testing different solutions I decided for the simplest solution there is. It’s not the best one, mind you, but it’s simple and efficient. You do not need to get your own api key and its only a few lines of code. The solution is launching the web browser with some parameters, one of them is the string that will be tweeted.

public void twitterAction()
	{
		String yourTweet="This is your tweet!";
		String url = "https://twitter.com/intent/tweet?text="+yourTweet+" %2simple";
	    Intent i = new Intent(Intent.ACTION_VIEW);
	    Uri u = Uri.parse(url);
	    try
	    {
	    	i.setData(u);
	    	startActivity(i);
	    }
	    catch(ActivityNotFoundException ex)
	    {
	    	Toast.makeText(getActivity(), "No browser installed", Toast.LENGTH_SHORT).show();
	    }
	}

After calling this method, your browser will be opened and you will have to login. Afterwards your message will be tweeted along with a hashtag. If there is no browser present an exception is caught and displayed with a toast message. Simple and nice :).

How to export an Android application

By following this guide, you will learn how easy it is to export any installed app on your Android device. Why is this useful, you ask? Well, I can think of two good reasons right now. For example, if you would need to decompile an apk, you would need to export it first. After that, you can do anything you want with it. The other one is even more interesting. When you buy a app, you have 15 minutes, to try out the app and then refund it. Many things can happen in these 15 minutes :).

First off, you will root access to your phone. If you do not have it and want to take care of that yourself, I recommend guys at XDA developers. They know their stuff and you will most definitively find a good tutorial on rooting your phone over their forums. The second one, you need to install the Root Explorer application. The free version is not good enough for this tutorial. After this its smooth sailing from now on.

export an android application

Download your desired application from Google Play and wait for it to be installed. Afterwards start up the Root Explorer. You will be prompted for root access, which you must grant. Tap the data folder, then app and you will get a list of all installed applications. Long press on the recently installed application and choose copy. Go two folders up, scroll down to sdcard and tap on paste. This will copy the apk file on your sd card. If you do not have one, then copy it somewhere else it does not matter where exactly. That is all there is to it, exporting an android application is really simple.