Navigating the Knowledge Landscape

Most of us are struggling with navigating the web. I know I am. The constant feeling of information overload and decisions about which path to follow. All while trying to keep up with the big picture, not lose sight of goals and find something new and interesting. Me and my friends have been exploring ideas on how to solve this*. We believe the solutions are in how we navigate our physical environment.

The prerequisite for successful navigation is a map and locating yourself on the map. Luckily the field of science mapping is blooming** and offers solutions for how to represent the vast knowledge networks out there. Below I've created an idealized 2D illustration of the entire map of science.

cience_illustration_thumbnail.png
Seeing the similarities between the map above and Earth Atlas is quite straightforward. For exploring knowledge landscape we need similar solutions as are available today for navigating Earth's map: tracking the path, planning the route, flying over the map and most importantly, zooming between different levels of details. For a hint of the power of zooming, try using your browser zooming in the image popup window above. Compare the speed of zooming to navigating the web slowly via links. In the final solution, the images will need to be filtered when looking at higher abstraction levels.

For getting a feeling of vital navigation features - Erki has been developing a visual browser for Wikipedia called Wikinity. In the video below you can see a demo of features for the early version of Wikinity. Together with zooming, this is how navigating the web could feel like in the future.



The future improvements list is long and I'll just list two that relate directly to the rapidly changing nature of knowledge. By making "tectonic landshifts" on the entire landscape visible, we can refer back to our previous understanding and reflect on the changes. By highlighting interesting areas of activity and zooming into the details, we can see what really goes on there. No more feeling anxious about the changes or falling into a link frenzy to find what you need. It will be right there, just zoom in.

Text is so last millenium. Knowledge maps are the future.

[*] Special kudos go to Silver, Allan, Ando, Taavi for their ideas on navigating the invisible.
[**] Katy Börner's book "Atlas of Science" triggered a lot of ideas.
Enhanced by Zemanta

Mapping Personal Contacts

| Comments (2)
contact_distribution.pngI've been having increasing number of discussions about how our behavior is affected by geographical and temporal spread of social, technology, concept and resource networks. Following is an exploration of my own social network's geographical spread.

I exported my Facebook, LinkedIn, Skype, Twitter, sent e-mail and travel contacts. Merged and augmented the lists with location information while subjectively filtering out contacts that were purely transactional (e.g. a single e-mail on a quickly decaying topic). This part took about 80% of the time - which is the case for most data analytics efforts today. However I believe this will change as the datasets become more open. For getting the city coordinates I used a script someone had created earlier. The small image in the upper right corner (created in R) gives an overview of how my contacts cluster into different cities.

Armed with location based contact lists, I fed the data to Python's Basemap module and voila, below you can see where my social network is located. The larger your virtual footprint, the more accurate it will be. Because of the distribution of my contacts, I used logarithmic scale for sizing citypoints - this makes the effect of my employer's offices (e.g. Tallinn, Stockholm, Prague, Luxembourg, London, San Fransisco) more evident.

europe.png
world.png
The visualizations could be further improved by using heatmap, plotting interactions between groups (requiring collection of further data) or by visualizing the map in time.

The move to virtual communication channels opened up the possibility to explore how individual's behavior is affected by the neighbors. The Internet is teeming with various analyses and initiatives like FuturIcT are being started to explore the techno-socio-economic dynamics giving us a better sense of how we, as a collective system, behave. With data about individual social networks expanding, the methods provide meaningful results also when analysing personal egocentric networks.

Open Up Public Data

I wrote this text initially in Estonian and then translated with Google Translate modifying the result here and there.
Estonian version is available in Finantsvaade.

Already 20 years ago Robert Reich drew attention to the increasing role of "symbolic analysts" - the increase in the proportion of people whose profession is to simplify reality or to reorganize it (e.g. engineers, teachers, scientists, journalists). Today's Internet penetration and the growing amount of data is making this new reality tangible to nearly 2 billion people. Information overload and it's diversity is reaching levels where it will test anyone's attention and decision-making capabilities. Who hasn't come across analysis paralysis?

digital-universe-2010.png
In the society of symbolic analysts, informed decisions are becoming a scarcity and given the speed at which data continues to grow (by 60% in 2009) we need solutions that automatically process, analyze and share the data. Outside of Estonia this topic is receiving a fair share of attention to the extent that visualzing data is becoming a profession (e.g see New York times infographics or UK's public data). Estonia has an advantage of being nimble due to our size and also benefits from highly integrated public systems.We must build on our strengths and open anonymous machine-readable data to everyone so that informed decisions can become the foundation of our society.

Obviously, good decisions alone are not enough - the pressure to move fast requires excellent execution. The good news is that methods for rapid and large-scale infrastructure changes have been developed for years in private sector. For example, JIT and Kanban in automotive industry allow cars to be built in days while minimizing waste. Software development sector has found inspiration in this and has evolved agile management methods enabling companies to adapt to the rapidly changing market conditions.

Estonia has a unique opportunity due to it's size and age to become a fast moving country if we open up anonymous data for machine-readability. As a result, explosion in data analytics will allow for significantly shorter strategic discussions and together with nimble management methods will also simplify prioritization.

Enhanced by Zemanta
Since my late interests have been related to networks, I figured that starting pet projects would help me to understand and share ideas about the field faster. Following is a short report describing explorations of natural numbers network. I chose this network because compared to real-world networks that are in my focus during the daytime, numbers network has been analysed for centuries and I hope that new ideas will emerge from this interaction. I acknowledge that for given topic, using number theory would be more appropriate, but hopefully you will find something interesting here - your feedback and ideas are very welcome. Please note that I do not have background in number theory.

The size of the network explored was largely determined by the available hardware and tools used (Gephi, NetworkX, Python and R). Larger networks could be explored with alternative libraries like Snap.

Generating the Network
Study of networks is a rapidly emerging interdisciplinary field, related to well established graph theory, that focuses on properties of real-world networks (e.g technological, biological, social, information, economy, etc). Since network theory offers new perspectives on understanding connections between objects and dynamics of highly interconnected systems [1], it should be an interesting exercise to take a closer look at relationships between the natural numbers.

To generate the numbers network, each number is considered a node and edges between them are created through factorization into primes and division into composite numbers. Self-loops are excluded, but connection to number 1 is kept for powers of primes to create a network with a single component. I was pleased to find out that related experiments have been done with subsets of the natural numbers and as a response it has been argued that analytic number theory is more appropriate as it shows the properties of the subset more directly [2]. {These analytic results can be used as a guide in later explorations of the methods.}

G = (V, E')
V = {1, 2, 3, ..., n}, n = 2^x, x in [2..19]
E = {(a, b): a | b or b | a}
E' = {(a, b) in E: a > 1 and b > 1, or a is a prime power, or b is a prime power} \ {(a, b) in E: a = b} [3]

network.jpg
The figure above visualizes the network structure of the first 1024 natural numbers. The nodes are colored according to their degree (number of edges the node has) and the higher the degree, the darker the node. Some of the trivial properties of the generated network are clearly visible - powers of prime with high degree and clusters of primes (on the right) being pulled into the tightly connected core of the network. The layout of the visualization is based on a force-based algorithm called Force Atlas where edges attract nodes to each other similar to springs and nodes repel from each other similar to electrically charged particles (the actual algorithm is a tad more complex; explore Gephi's internals for details).

Taking a look at the log-log scale diagrams of 0,5 million numbers below, the difference between the degrees of powers of primes and non-prime numbers is evident. Overall the frequency of degrees in the analysed network exhibits some regularity {and hopefully someone with proper background in number theory can point out whether this is related to the Dickman distribution or something else}.

degrees.jpg
Network Evolution
After a quick look at the static topology of the network, delving into exploring the dynamics of growth is the next obvious step. The approach used here is based on network snapshots, where properties are calculated for each snapshot. The calculations were made for networks sizing to powers of 2 up until the network of 131 072 numbers. Connectedness and components are not calculated as the network generated is limited to a single component.

One of the findings during the simulation, not show here, is that the network diameter (greatest number of edges in shortest paths between two nodes) remains 4 since new products of unique primes will require four hops to reach each other.
edges_avg.jpg
As the size of the network increases, the growth of edges, average degree and average path length seem to follow an asymptotic increase. While the initial attempt with linear regression for edge growth gives a good approximation (log(edges) = 1.1590 * log(numbers) + 0.2125 with R-squared = 0.9973), a closer look at the fit suggests that different models must be sought. As pointed out in [2], analytical approach can provide the best guidance for finding the right models, since the dynamics is related to the probability of two numbers being coprime (numbers that share no common factors except 1 have a probability 6/Pi^2 over all primes). As network evolution deals with finite sets of nodes, taking a look at prime counting functions is also required. {Describe the resulting growth models.}

Most of the properties presented above are calculated with simple counting algorithms, except the diameter and average path length that are based on calculating the shortest path lengths for all nodes with breadth first search traversal. Below, evolutions of other common properties are presented.
cluster_density.jpg
Density (on the right chart) represents how far the numbers network is from being a complete graph, where each node would be connected by a unique edge (density=1). Density is calculated trivially based on the number of nodes and edges in the network. A related measure called average clustering coefficient (on the left chart) shows how many of the edges tend to cluster together. It is calculated by finding the number of existing triplets for each node and is used to understand the level of transitivity (closed triplets) in a network, frequently observed in real-world networks, as compared to random graphs where edges are generated randomly. As an example, clustering coefficients of two random graph models (with similar densities to number network snapshots) are presented for comparison. Note that for Erdos-Renyi model, the resulting graph is not connected and for Watts and Strogatz, the connected small-world graphs are generated through repeated generation.

Additional measures like modularity could be used to explore community structures within the network (e.g. based on node degrees) to understand assortativity, but this is not elaborated here since we are not exploring a real-world network.

Summary
I hope that this short exploration of natural numbers with network analysis methods proved interesting even without proper number theoretical explanations. If you are well versed in number theory, then hopefully it allowed to gain some insight into how network theory can be applied or how it could be improved. In case you had any ideas while reading this, then please leave your feedback below in the comments.

There are many alternative ideas that could be explored; to brainstorm a few: analyze a weighted graph of prime numbers based on powers of factors; explore percolation by removing numbers from the network. However I'm somewhat reluctant to venture deeper without receiving "sanity check" feedback.

Update: Some feedback in Physics Forum and Math Forum.

[3] CRGreathouse in Math Forum.
Enhanced by Zemanta

Lessons Learned

The Bank of England in Threadneedle Street, Lo...

Image via Wikipedia

While browsing through old notes about my financial behavior, I stumbled upon a short list of statements that I wrote down about things I've learnt through trial and error. It is interesting to see how much effort it takes to follow these ideas even though they are common sense. Overall I am satisfied with the adaptions I've made during the financial turmoil, but there is always room for improvement especially when looking forward into the uncertainty. 

Here are the lessons learned:
  • Closure! Once a decision is made and no new information has become available, don't stall, but finish it. Execution and closure have been the most critical factors when it comes to successful decisions. The price of no decisions can be very hefty.
  • Trust your intution that is based on facts.
  • Don't rely on emotion in big decisions - they are too complex. Make sure you take time out to understand whether you are rationalizing, dreaming, drooling, fearing.
  • Stay active - mentally, physically and professionally. Without this you won't be able to process new information leading to good decisions or new ideas leading to good opportunities.
  • Don't be afraid of risk as it is there in every decision. Instead, take a calculated risk - put risks into numbers as emotion will just get into the way with truly complex opportunities.
  • If you want to gamble, go play small-stakes poker. High-risk does not mean gambling. Read the points above!
  • Participate in public market investments, this will ensure that you are aware of the trends and status quo. Without this context, the facts will not appear to be tangible. However it takes significant and continuous effort to get positive gains, so use index funds instead of single companies if all you can spend is a couple of hours on Sundays.
  • Choose your partners carefully as you'll potentially have to spend many years together with them.
  • Time your investments. If you think that the market is too high, but can't wait, do the purchase in reasonable chunks.
  • Going short is only good if you know what you are doing. It is riskier because the energy of the companies against whom you are betting is usually spent against the decline.
  • Have a few exit strategy scenarios for personal ventures.
  • Changing top-management takes a lot of dedicated effort, which you have to plan for, but most importantly you have to executed flawlessly.
  • Know when to quit - if you see that the high-growth period is over for good and your interests have shifted, either find a way to grow in a new direction or leave. Don't stagnate.
  • Take the initiative and don't give away your shares too easily as options will do miracles.
  • Consider the cost of your own money, loss of potential income and price of your time when making a new venture. Is it a hobby or a business you are starting?
  • Be aware of your own behavior:
    • hindsight bias - did you really see it coming; it does not mean you see the future.
    • anchoring - past performance is not a good prediction of future performance. Refresh your variables and take a fresh new look.
    • confirmation bias - accepting information that confirms initial decisions is always easier. List your assumptions, both positive and negative, and keep them up-to-date. This will help you notice when the tide is turning.
    • recency bias - looking farther into the future involves higher risk, thus situation today blinds us from the megatrends. Refresh your variables.
    • adjustment bias - using minimal corrections to avoid countering the past decision. Make your stop based on assumptions today and stick to it in the future. If you instead consider to buy, refresh your variables before!
    • self-attribution bias - you are usually not in the control of your investments.
    • herd behavior - following others; avoid this by making it explicit why you are making a different decision.
    • gambler's fallacy - going against the trend too early.
Enhanced by Zemanta

About Decisions

Picture 66.pngAmygdala and the left frontal lobe are actively involved when humans are making decisions. It is possible to damage the brain so that people will be unable to make decisions without auxiliary. The hypothesis is that there are also people who are extremely quick at decision making.

TEDTalks: Technology and Education

This is part 3 of an unfinished stream of ideas on various topics combined with my own interpretations.

One could argue that we are on the brink of technological singularity - adoption rates of new inventions are increasing, technologies are improving with exponential growth and some have predicted that by 2010 the amount of technical data will double in 72h (compared to 2 years in 2007). There is little reason to doubt that we will use our newly found knowledge to improve human body, fight diseases and improve our brains that are still in beta mode. If you are a techno-pessimist, read Singularity Sky by Charles Stross - fear is just an intermediate state before techno-utopianism kicks in.

Of course exponential growth has always certain assumptions that are hard to fix in real life, like the energy usage, but it seems that extensive media coverage of environmental issues has forced companies to mobilize (e.g. Walmart saving 10% of electricity by just having buildings with glass ceilings) to free up the energy for more important stuff like CERN :) So lets assume that singularity is here to stay. What would be the role of education in a world where change happens faster than our current carbon-based bodies can handle?

In a world where everything is uncertain, one has to tap into the creativeness, and children have unlimited capacities for innovation. It's about time to start treating creativity with the same status as literacy. Kids should be allowed to take chances - if you are not prepared to be wrong, you won't come up with anything original. Right now, we are educating people out of their creative capacity.

Next topic please. Nature.

TEDTalks: Design

This is part 2 of an unfinished stream of ideas on various topics combined with my own interpretations.

chair.pngHumans crave for complexity, we enjoy and consume the things that fulfil this interest, we want less of things that are routine and simple. For a happy life we look for things that create more enjoyment and less brain. We get thrilled by mystery boxes - a closed box has infinite possibilities, giving us hope that it has things in it that bring us enjoyment. To be remarkable, target the geeks and early adopters with your mystery box design and hope that maybe they will spread the word. Reveal it partially. In addition, you won't know what will work, so don't bother making it perfect the first time.

If you have to work with a designer, then keep in mind that there are three types of designers:
  • cynical designers, design for marketing (see Microsoft's iPod packaging);
  • fantastic designer, design for other fantastic designers (yes, that's a chair on the picture). These are the ones who get hit first during a recession;
  • sustainable designers, design for human being, for society - cradle to cradle. They are not the architects of American suburbs.
However, technology creates a blank page and creation of media becomes ubiquoutus and democratized.

Next topic please. Technology and education...

TEDTalks: Mind

Over the years I've watched +70 different TEDTalks, while making 1-2 sentence summaries of each. As I feel a bit saturated, it's time to organize the old notes and make room for new ideas. This is part 1 of an unfinished stream of ideas on various topics combined with my own interpretations.

big_thoughts.pngBrain, the home of the mind, surges in performance starting from puberty until the early 30ies, from there on it becomes pretty much stable until the early 60ies, when aging starts to kick in. Brain is built up from two hemispheres: left side works like a serial processor - it is methodical, reasons about the past and the future, categorizes knowledge, thinks in language; right side works like a parallel processor - it's concerned about right here and right now, analyses stimuli present at the moment. Without the left side one lives in a lala-land, free from concerns, so if you want to relax just use your right side. It is interesting to note, that TED presentations involving music, dance and poems don't have any notes - almost as if my left side stopped working.

The connectivity diameter of the human brain is only five or six neurons if you view it as a graph, which is similar to small-world networks present in social networks, connectivity of the Internet and gene networks - would the breakthroughs in one area lead to revolutions in others?

Mind has a recursive peculiarity called self-awareness (a novel Understand by Ted Chiang explores this aspect to the extreme). Everyone feels like an expert about consciousness, however unless you work as a scientist in the given topic, your insights about it are most probably wrong due to the small sample size. These intuitive incorrect assumptions about the internals of the mind and brain are holding us back. One of the major misconceptions is that intelligence is defined by behavior, however very complex behavior can be a result of simple rules; Conway's Game of Life is the classic example of this statement. The correct theory would be that memory prediction leads to intelligent behavior - theory of memory is about patterns, sequences, auto-associations, predictions. Creative people have stronger connections between voice, touch and vision parts of the brains - what about people who don't have any vision or hearing? Reading a bit about Helen Keller might provide the insight.

Our minds interact and our body just hangs along with the way. Language emerges from this interaction giving us a window onto human nature. Human intelligence is expressed through concepts and metaphorical abstractions of those concepts. Indirect speech is effective in negotiation as they avoid direct attack on partner's beliefs, such mismatches create awkward moments. Similarly, we would also recognize Martian letters for K and B, if these would represent similar forms in Martian language. As the complexity of our thoughts is dependent on the environment our mind lives in, are we be able to reason what would happen if our bodies would be replaced by something else, e.g. digital body with superior sensing capabilities? This reminds me that I need to read Altered Carbon by Richard Morgan.

Next topic please. Design...

Edit: An interesting discussion about recursion and human thought by Daniel L. Everett.

Art of Start by Guy Kawasaki

| Comments (2)
Watched a great video about startups, here are the bulletpoints:

1. Make meaning
  • increase the quality of life
  • right a wrong
  • prevent the end of something good
  • money follows... avoid MBA's and consultants in the beginning:P
2. Make mantras (for employees)
  • avoid mission bullshit
  • 3 or 4 words ("just do it" is for customers, "authentic athletic performance" for employees)
3. Get going
  • think different (don't do 10% better, do 10x)
  • polarize people (perfect product for everyone will create mediocracy)
  • find a few soulmates (people to balance your weaknesses)
4. Define a business model
  • be specific (facebook is unusual)
  • keep it simple (ask feedback from women :))
5. Weave a MAT (milestones, assumptions, tasks)
  • milestone (prioritize; shipping is important)
  • write down and test your assumptions
  • tasks (PPP)
6. Niche thyself
  • not unique, high value to customer (compete on price; ok, but not good)
  • no value to customer (stupid)
  • no value, not unique (dotcom stupid)
  • unique product, high value to customer (print the ticket at home :P; marketing)
7. Follow the 10/20/30 rule
  • 10 slides in powerpoint in 20 minutes with 30 font
  • title, problem, solution, business model, underlying magic
  • marketing and sales, competition, team, projections, status and timeline
8. Hire infected people (workexperience, education)
  • ignore irrelevant and hire people who love your product
  • hire better than yourself
  • apply shopping center test (like the person)
9. Lower barriers to adoption
  • flatten the learning curve
  • don't ask people to do something that you wouldn't
  • embrace your evangelists
10 . Seed the clouds
  • Let a hundred flowers blossom (take the money from wrong customers :)
  • Enable test drives
  • Find the influencers
11. Bonus: Don't let the bozos grind you down
  • can't work, won't work, don't need, etc - ignore
  • ignore the high authority bozos