Analytics Tools

posted 22 Jan 2013, 07:36 by David Sherlock

Putting CSV into a triplestore

posted 6 Nov 2012, 03:55 by David Sherlock

Rough, missing bits

YouTube Video

Community Detection- facebook friends

posted 12 Sep 2012, 06:13 by David Sherlock

Video to remind myself

Buergschaft4 Processing

posted 18 Jun 2012, 07:53 by David Sherlock

Had a go with Buergschaft 4 in Processing using my comments on the jisc cetis blog. Realised I should have taken stop words out. Screenshot of Java applet

Coronation Street marriages

posted 18 Jun 2012, 04:27 by David Sherlock   [ updated 18 Jun 2012, 05:48 ]

Failed R experiments #29123

posted 2 Jun 2012, 08:32 by David Sherlock   [ updated 2 Jun 2012, 08:41 ]

R is a difficult place to start learning something new. I’m currently writing a beginners guide to Complex Networks and made the mistake of going down the route of learning some of the basics in R. As every programmer knows you only real learn through failing and documenting them is just as important (if not more important) as documenting your success.

Problem: Had a two column dataset that I wanted to turn into a one mode model graph in Complex Network analysis software. Attempted to do it in R. Thought I would start with something visual.

This was the code:


projectDir = getwd() 

codeDir = file.path(projectDir, 'code')

dataDir = file.path(projectDir, 'data')

outputDir = file.path(projectDir, 'output')




  print("Verbose SET")


organisations_events<-read.csv("/Users/David/Desktop/PhD/R_github/ROI/data/Ins_Event.csv" , header=T, sep=",")

df<-read.csv("/Users/David/Desktop/PhD/R_github/ROI/data/Ins_Event.csv" , header=T, sep=",")

M = as.matrix( table(df) )

Mrow = M %*% t(M)

Mcol = t(M) %*% M

iMrow <- graph.incidence(Mrow, mode=c("all") )

V(iMrow)$label = V(iMrow)$name

V(iMrow)$label = V(iMrow)$name

V(iMrow)$label.color = rgb(0,0,.2,.5)

V(iMrow)$label.cex = .4

V(iMrow)$size = 6

V(iMrow)$frame.color = V(iMrow)$color

 plot(iMrow, layout=layout.fruchterman.reingold)


 This was the result:





In fairness I had created two relationship matrixs correctly, which was an achivement for me myself:

M = as.matrix( table(df) )

Mrow = M %*% t(M)

Mcol = t(M) %*% M

Maybe I should jump ship to another tool from that bit. Perhaps some help here:

Quick thoughts on

posted 23 May 2012, 06:15 by David Sherlock   [ updated 24 May 2012, 07:29 ]

Today I had a poke around, which had somehow managed to escape me until now. Socl is a social network project developed Microsoft, but with G+, Facebook, MySpace and Twitter all fighting for your time how can you justify yet another time sink? I created an account, had a poke around and then deleted my account.

As Moot recently highlighted, a problem with Facebook and G+ is that they are identity-based services making a huge presumption about a persons identity; that it boils down to a name and a picture. Socl is an attempt to make social networking to be more interest based.  It promises to help you 'discover new topics, express yourself and learn'. It seems in socl your identity is built on web searches, you search for things you like, share them and then your friends can ‘riff’ (whatever that means) on them, videos around your interests pop up and discussions around the interest can take place in video parties. It’s all very confusing. I would hope that Microsoft have thought really hard about identity and what the next generation of social network should look like but in reality I think Microsoft have thought really hard about what will turn advertisers on. 

Playing with Attachment Simulation

posted 22 May 2012, 08:24 by David Sherlock   [ updated 23 May 2012, 01:22 ]

Early Work

Experience and knowledge through observation

posted 16 May 2012, 03:13 by David Sherlock   [ updated 24 May 2012, 07:35 ]

(This was my 2nd attempt. I'm still not happy with this, but its getting there.)

I find the area of learning through play fascinating, as Isenberg and Quisenberry (2002) famously once said

“Play is not wasted time, but rather time spent building new knowledge from previous experience”

Quite often the links between play, learning and building new knowledge is made in the context of the development of young children, as play is a safe environment to build new experience without having to be in the actual context that is being emulated. These links are not restricted to young children though; there are examples in nature such as baby lion’s play fighting to learn hunting techniques, which their future survival will depend on. Recently, due to developments in processing power there have been movements to create computational simulations of social situations that are created by emulating the participants of that situation. By being able put ourselves in the position of a participant of the situation the simulation is giving us the ability to gain knowledge and potential experience without having to physically there. There are many reasons why we might not want to be part of the situation, for example it may be dangerous, historical or simply theoretical.

I find that this style simulating social situations are being created two different domains, with the different domains having a different ‘end goal‘.  The gaming domain has the end goal of being fun, while the second domain Agent Based modelling has the end goal of being realistic. However a 3rd  domain, which is a crossover of both domain is starting to emerge. 

Simulation through Gaming

Social simulation is not new to the gaming community , games such as SimCity have long allowing players to ask questions around the experience of developing a city. What would happen if an airport goes here or taxes are reduced there? More recently games such as Dwarf Fortress combine city building elements with the interpersonal psychology of individuals that allow users to explore how treating these individuals has an effect on the well being of the social fabric as a whole. 

Agent Based Modelling

As Wikipedia tells us, Agent-Based Modelling is a method of creating models to simulate the actions and interactions of autonomous agents. This is atechnique that has been adopted by the social science community to simulate social systems to assess what can change the behaviour of agents that often represent people and how these changes will have an effect on the system as a whole. This domain not so much interested in fun for the end user but in validation of the model so that statements can be made about the system it is representing.

Traffic Sulation in netlogo

Using a freely available tool to create a zombie simulator

A 3rd Domain?

I identify that a middle ground between the two domains is emerging, in part due to the recent emergence of low-entry level tools from the 
ABM domain and low cost development kits.

Using these tools it is possible to create simulations where the user is able to envision themselves in a social system without being there. The process of creating the model is part of the experience in itself, with the developer thinking through the make up of the agents, the variables that will effect these agents and about the environment that make up the system. Running the simulation gives the developer an insight into how those individual agents make up the system as a whole and what small changes on the micro level might look like on the macro level, giving them an insight into what they could change it before it is run for a second time. There is no falling in love with data in this domain as the primary interest is in having fun (and as a result learning from experience) but it uses the ABM techniques of creating simulations at the agent level.

Untitled Post on ABM

posted 15 May 2012, 07:07 by David Sherlock   [ updated 15 May 2012, 07:08 ]

How interactive visualisations, agent based models and social simulations give us experience through observation.




In an Agent Based Models you create a system by programing individual autonomous agents. The idea is when you press the play button the individual agents start to behave in the way you programmed them, the actions of each individual agent have an effect on each other and the system as a whole.  

I like this approach when applied to social simulations because I feel it gives an insight into the happenings at different levels of society, all the way from the personal psychology of the individual to the social science behind the behaviour of group as a whole.

My thought process when developing an agent based model is split into 3 sections:

Step 1: (Programming) The individual 

I write some code to create agents. 

Who are the key players in this simulation? 

What makes up their personal psychology? 

What is their immediate environment like and how do they respond to it? 

How will react when they interact with other agents, what will the effect be on the environment?

What are the changeable factors?

Step 2: (watching) The interactions between agents

I press play. 

How do the agents react to one another, as suspected? 

How have they changed during the interactions? 

How do they behave after the interactions? 

What effect did this have on the environment?

If I change variables that effect the environment or the agents, how does this change the simulation? 

How is what is happening at the social layer effecting what is happening to the individual agent?

Step 3 (reprogram)

Did the simulation mimic real life events?

Reprogram agents/environment to see if I can mimic real life more accurately.

Add changeable variables

Restart Program

The questions and process that I am going through are naturally an important part of developing agent-based models, but these questions are the same questions we should be asking when we evaluate what is happening in society. 

Say that I created an agent-based model around the 2011 England Riots, or more preciously the events leading up to the riots   How would I answer these questions then?

Step 1: (Programming) The individual 

Who are the key players in the riots simulation?  Rioters, police, family, society, gangs?

What makes up their personal psychology?  What about their morals, beliefs, education, hopes and dreams? 

What is their immediate environment like and how do they respond to it? Both physical and social. Where do they live? Who do they live with?

How will react when they interact with other agents, what will the effect be on the environment? Depends on the make up of the agents. Will they riot?

Step 2: (watching) The interactions between agents

How do the agents react to one another?  If negatively, what are the variables that change how they interact? What are the methods of communication?

How have they changed during the interactions?  Are there patterns in rioter behaviour that can be identified early on? How do you disrupt the patterns?

How do they behave after the interactions? Is their behaviour influence to other agents, what are the environmental and agent parameters that make them copy behaviour of act a certain way?

What effect did this have on the environment? Can the environment changes have an effect on the likelihood of an agent to riot?

If I change variables that effect the environment or the agents, how does this change the simulation?  Would a better education, family life, environment, penal system, etc effect the agents enough to stop a riot breaking out

How is what is happening at the social layer effecting what is happening to the individual agent? Is there a tipping point? 

Step 3 (reprogram)

Did the simulation mimic real life events? 

Reprogram agents/environment to see if I can mimic real life more accurately.

Restart Program

Even a very basic England 2011 riot simulator would get me to think about the agents, environments, communications and factors involved in simulating a riot and the relationship between them. Agent based modelling, however crude the model, gives us experience through observation.  

Simulating Society’s

When creating models to simulate a social setting we are giving people a chance to experience the setting through observation and reprogramming of the model. To achieve this the model does not have to be perfect and we can avoid the pitfalls of falling in love with data.

I think this is a very powerful thing to be able to do, but my love for ‘experience through observation’ does not come from agent based modelling community but from the gaming community. I often find that the Agent Based Modeller is trying to perfectly replicate a situation, which is not my primary interest.

My introduction to simulated society was 

No model is perfect 

I think the key is not to fall in love with data

1-10 of 42