OObject the latest Wists site launches today.

Oobject is a gadget blog, but with a difference. Instead of posts, there are ‘charts’ constantly updated image galleries with the best items in each category, voted on by users. As usual, the focus is or quirky, unusual or well designed lists of things.

OObject is a bit like Billboard charts for gadgets.

OObject is a major leap forward in terms of the way our sites work as it completely couples the wists publishing system into a customized version of Wordpress, and uses the new editorial back-end of wists for management.

How to setup a cheap Infinite Disk system using S3 without having to use EC2.

I don’t normally post technical stuff on my blog, but I’m putting this up because I think it might help others.

Wists images are stored in the file system with filenames based upon a hash of their url (principally). This keeps the db small (overloading the directories is prevented by creating subdirectory structures based upon the first few letters of the hash (/b/c/bc267867 etc.)

We want to use Amazon for reliability and unlimited storage, but we need to couple the system to application logic (and our code isn’t multi-treaded), so a straightforward S3 use isn’t possible.

The problem with S3 at the moment is all to do with latency. We don’t want to have to manipulate and store wists thumbnails remotely on an EC2 instance, but if we do it locally then there will be a delay sending the images over the wire and storing them in S3. Furthermore, rsync or rsync style approaches are a real load problem for us so we have to do batch backups every hour based upon the age of files (in theory rsync should be lighter weight, in practice it is not for our particular use). We also use cache.wists.com via a CDN that pulls the images when they are there, using squid. We will keep this piece, because the CDN works out cheaper than S3, but it can’t readily be used as a backup service, however the method below does not depend on it.

Step 1. Setup a cron to call a shell script hourly, that backs up files to S3 that are more recent than 1hour, and deletes files that are older than 1 week (we will keep a weeks images as a buffer in case there is a failure).

Step 2. Point the CDN to S3 (instead of at Wists servers).

Step 3. Change application code so that all image requests are for local local versions, instead of directly from the CDN i.e. wists.com/images/foo (instead of from cache.wists.com at moment), the application logic will check to see if the exists, if not they try cache.wists.com and lastly show a placeholder as default if this in turn doesn’t exist.

This way all files newer than 1 week will be served live from wists and everything else from the CDN which will cache from the S3 backup.

Although this usage is perversely the reverse of the normal way a quid based CDN is used (i.e. CDN usually checks if file exists on each call, and then caches, if not) and this will result in higher webserver loads, the advantages of reliability and simple infinite storage outweigh the disadvantages in our instance.

Amazon S3 PHP Class Update - neurofuzzy, flash game development, rich internet applications, free source code - *alt.neurotica.fuzzy*

Atheist Media Blog

“high levels of organic atheism [not government coerced] are strongly correlated with high levels of societal health, such as low homicide rates, low poverty rates, low infant mortality rates, and low illiteracy rates, as well as high levels of educational attainment, per capita income, and gender equality.”

Interestingly, the US is 4-5 times more religious than Israel.

Atheism: Contemporary Rates and Patterns

According to the Global House Prices blog: “At the peak of the Japanese real estate bubble, the Imperial Palace in Tokyo was worth more than the entire state of California”.

Looking at the graph of US and Japanese Real estate prices is sobering, the US one is like a ski slope and the Japanese one, a mountain.

But none of this seems to make sense to us Brits. Housing in America still seems cheap, by UK standards, even in New York. Consider that the average house price across the whole of the UK, not just the expensive bits, is over $400,000.

The fact that I could barely afford to buy back the building that I bought in my 20s, despite having earned much more than most, since, means that something is fundamentally out of whack in the UK and by an order of magnitude that is much larger than in the US.

And it’s not just housing. When I was last there I bought a glass of wine in a pub in a smallish town several hours drive from London. It was $17. A drink in a five star hotel in the center of Manhattan is cheaper.

Yet despite this, when I go back to the UK, this is what I hear “house prices may stop rising, but they’ll flatten out, they never go down”. Then I get a lecture about how stupid I was to have sold.

People in the US are waking up, while those in the UK are deluded.

Global House Prices: Housing Bubble Facts and Figures

Wists users continue to rise, and monthly uniques are around the 350K mark, but our Alexa is dropping and deliberately so.

The reason is that we have decided to ruthlessly cull spam users, which account for a large number on many social shopping sites and social networks.

Social network spam doesn’t tend to be that visible since the items on front pages and showcased category links (top items etc.) of many social websites are hand picked by editors or users. But do a search for high value Adsense terms within many user generated sites and you will see that the automated publishing tools have blasted them, creating useless ballast that companies are reluctant to get rid of since it generates organic search engine traffic, and makes the numbers look good.

Since the type of spam we get is SEO spam and a greater percentage of these people use Alexa toolbars , this is a risky thing to do for your ‘industry’ image, since it lowers your numbers - but we think it is the right thing to do in the long term, since it obviously makes a better product for users.

Social shopping is extremely prone to spam, for obvious commercial reasons, do a search for Viagra on Kaboodle, for example, and ponder as to how many of those users that Hearst just bought are useful. Of course I shouldn’t single out Kaboodle, they deserve congratulations on their acquisition and their technology is very solid, but they show the considerable pressure these days shared by many Internet startups to quote raw numbers and eyeballs, which sometimes obscures true, long-term value.

After the dot-com crash, the somewhat obnoxious phrase ‘eye-balls’ seemed to disappear. It’s back, but with Wists we are ultimately only interested in what people see, not how many people are looking.

Wists

Posted in diary | Comments Off