Benford's law
- For the tongue-in-cheek "law" about controversy, see Benford's law of controversy.
Benford's law, also called the first-digit law, states that in lists of numbers from many real-life sources of data, the leading digit is 1 almost one third of the time, and larger numbers occur as the leading digit with less and less frequency as they grow in magnitude, to the point that 9 is the first digit less than one time in twenty. This is based on the observation that real-world measurements are generally distributed logarithmically, thus the logarithm of a set of real-world measurements is generally distributed uniformly.
This counter-intuitive result applies to a wide variety of figures, including electricity bills, street addresses, stock prices, population numbers, death rates, lengths of rivers, physical and mathematical constants, and processes described by power laws (which are very common in nature). Even more counter-intuitively, the result holds regardless of the base in which the numbers are expressed, although the exact proportions of course change.
It is named after physicist Frank Benford, who stated it in 1938, although it had been previously stated by Simon Newcomb in 1881 in his paper "Note on the Frequency of Use of the Different Digits in Natural Numbers". The first rigorous formulation and proof appears to be due to Theodore P. Hill in 1988.^{[1]}
Mathematical statement
More precisely, Benford's law states that the leading digit d (d ∈ {1, …, b − 1} ) in base b (b ≥ 2) occurs with probability proportional to log_{b}(d + 1) − log_{b}d = log_{b}((d + 1)/d). This quantity is exactly the space between d and d + 1 in a log scale.
In base 10, the leading digits have the following distribution by Benford's law, where d is the leading digit and p the probability:
d | p |
---|---|
1 | 30.1% |
2 | 17.6% |
3 | 12.5% |
4 | 9.7% |
5 | 7.9% |
6 | 6.7% |
7 | 5.8% |
8 | 5.1% |
9 | 4.6% |
One can also formulate a law for the first two digits: the probability that the first two-digit block is equal to n (n = 10, …, 99) is log_{100}(n + 1) − log_{100}(n), and similarly for three-blocks without leading zeros and longer blocks (in fact, the BL for the first p digits in base b follows immediately from the BL for a single leading digit in base b^{p}).
Explanation
The law can be explained by the fact that, if it is indeed true that the first digits have a particular distribution, it must be independent of the measuring units used. For example, this means that if one converts from e.g. feet to yards (multiplication by a constant), the distribution must be unchanged — it is scale invariant, and the only distribution that fits this is one whose logarithm is uniformly distributed.
For example, the first (non-zero) digit of the lengths or distances of objects should have the same distribution whether the unit of measurement is feet, yards, or anything else. But there are three feet in a yard, so the probability that the first digit of a length in yards is 1 must be the same as the probability that the first digit of a length in feet starts 3, 4, or 5. Applying this to all possible measurement scales gives a logarithmic distribution, and combined with the fact that log_{10}(1) = 0 and log_{10}(10) = 1 gives Benford's law. That is, if there is a distribution of first digits, it must apply to a set of data regardless of what measuring units are used, and the only distribution of first digits that fits that is the Benford Law.
More precisely, let X be a random variable whose probability of being equal to any positive integer, x, is proportional to x^{−s}, where s > 1. That is,
- .
The constant of proportionality must then be 1/ζ(s), where ζ is the Riemann zeta function (see zeta distribution). The probability that the first digit of X is n approaches log_{10}(n + 1) − log_{10}(n) as s approaches 1.
The precise form of Benford's law can be explained if one assumes that the logarithms of the numbers are uniformly distributed; this means that a number is for instance just as likely to be between 100 and 1000 (logarithm between 2 and 3) as it is between 10,000 and 100,000 (logarithm between 4 and 5). For many sets of numbers, especially ones that grow exponentially such as incomes and stock prices, this is a reasonable assumption.
Note that for numbers drawn from many distributions, for example IQ scores, human heights or other variables following normal distributions, the law is not valid. However, if one "mixes" numbers from those distributions, for example by taking numbers from newspaper articles, Benford's law reappears. This can be proven mathematically: if one repeatedly "randomly" chooses a probability distribution and then randomly chooses a number according to that distribution, the resulting list of numbers will obey Benford's law (Hill, 1998).
Applications and limitations
In 1972, Hal Varian suggested that the law could be used to detect possible fraud in lists of socio-economic data submitted in support of public planning decisions. Based on the plausible assumption that people who make up figures tend to distribute their digits fairly uniformly, a simple comparison of first-digit frequency distribution from the data with the expected distribution according to Benford's law ought to show up any anomalous results. Following this idea, John Nye and Charles Moul studied international macroeconomic statistics.^{[2]} They found that while most of them (e.g. World Bank international GDP data) conform to the law, it is not the case for GDP figures of developing countries suggesting a manipulation of the original data.
In the same vein, Benford's law can be (and is) used to analyse insurance, accounting or expenses data and identify possible fraud as well as pricing strategies (el Sehity, Hoelzl & Kirchler, 2005) .
Other uses, for example to analyse the results of clinical trials and election results, have also been proposed.
Limitations
Care must be taken with these applications, however. A set of real-life data may not obey the law, depending on the extent to which the distribution of numbers it contains are skewed by the category of data.
For instance, one might expect a list of numbers representing 'populations of UK villages beginning with A' or 'small insurance claims' to obey Benford's law. But if it turns out that the definition of a 'village' is 'settlement with population between 300 and 999', or that the definition of a 'small insurance claim' is 'claim between $50 and $100', then Benford's law would not apply because certain numbers have been excluded by the definition.
History
The discovery of this fact goes back to 1881, when the American astronomer Simon Newcomb noticed that in logarithm books (used at that time to perform calculations), the earlier pages (which contained numbers that started with 1) were much more worn than the other pages. It has been argued that any book that is used from the beginning would show more wear and tear on the earlier pages, but also that Newcomb would have been referring to dirt on the pages themselves (rather than the edges) where people ran their fingers down the lists of digits to find the closest number to the one they required.
However, logarithm books did contain more than one list, with both logarithms and antilogarithms present, and sometimes many other tables as well, including exponentials, roots, sines, cosines, tangents, secants, cosecants etc., thus, this story may be apocryphal. However, Newcomb's published result is the first known instance of this observation and includes a distribution on the second digit, as well. Newcomb proposed a law that the probability of a single number being the first digit of a number (let such a first digit be N) was equal to log(N+1)
The phenomenon was rediscovered in 1938 by the physicist Frank Benford, who checked it on a wide variety of data sets and was credited for it. In 1996, Ted Hill proved the result about mixed distributions mentioned above.
Popular culture
Benford's law was used as a plot device on CBS's TV series NUMB3RS in the episode "The Running Man".
See Also
References
- ↑ Hill, Math at GA Tech.
- ↑ John Nye & Charles Moul (2007). "The Political Economy of Numbers: On the Application of Benford's Law to International Macroeconomic Statistics". The B.E. Journal of Macroeconomics. 7 (1): Article 17. Check date values in:
|year=
(help)
- Frank Benford (1938). "The law of anomalous numbers". Proceedings of the American Philosophical Society. 78 (4): 551&ndash, 572. Unknown parameter
|month=
ignored (help) (subscription required) - Theodore P. Hill (1998). "The first digit phenomenon" (PDF). American Scientist. 86: 358. Unknown parameter
|month=
ignored (help) - Simon Newcomb (1881). "Note on the frequency of use of the different digits in natural numbers". American Journal of Mathematics. 4 (1/4): 39&ndash, 40. (subscription required)
- Hal Varian. "Benford's law". American Statistician. 26: 65.
- Tarek el Sehity, Erik Hoelzl and Erich Kirchler (2005). "Price developments after a nominal shock: Benford's Law and psychological pricing after the euro introduction". International Journal of Research in Marketing. 22 (4): 471&ndash, 80. doi:10.1016/j.ijresmar.2005.09.002. Unknown parameter
|month=
ignored (help) - Tarek el Sehity, Erik Hoelzl and Erich Kirchler (2005). "Price developments after a nominal shock: Benford's Law and psychological pricing after the euro introduction". International Journal of Research in Marketing. 22 (4): 471&ndash, 80. doi:10.1016/j.ijresmar.2005.09.002. Unknown parameter
|month=
ignored (help) - Wendy Cho and Brian Gaines (2007). "Breaking the (Benford) Law: statistical fraud detection in campaign finance". The American Statistician. 61 (3): 218&ndash, 223. doi:10.1016/j.ijresmar.2005.09.002. Unknown parameter
|month=
ignored (help) - L.V.Furlan (1948). "Die Harmoniegesetz der Statistik: Eune Untersuchung uber die metrische Interdependenz der soziale Erscheinungen". Reviewed in Journal of the American Statistical Association http://www.jstor.org/view/01621459/di985813/98p0504n/0?frame=noframe&userID=8614251252@york.ac.uk/01cc99331258e7114ffd0f8cf&dpi=3&config=jstor. 43 (242): 325&ndash, 328. Unknown parameter
|month=
ignored (help); External link in|journal=
(help)
External links
- Free Java Tool To Analyze Data Using Benford's Law by John Morrow
- Benford's Law and Zipf's Law at cut-the-knot
- Following Benford's Law, or Looking Out for No. 1
- A further five numbers: number 1 and Benford's law by Simon Singh
- A software which generates Benford subsets by Oktay Haracci
- A small flash application demonstrating Benford's Law, created by William Fawcett
- Looking out for number one by Jon Walthoe, Robert Hunt and Mike Pearson, plus Magazine, September 1999
- Template:MathWorld
- "Benford's Law" by Paul Niquette
- Statistics::Benford Perl module which calculates the deviation from Benford's Law
- Mystery of Benford's Law solved by DSP
cs:Benfordův zákon de:Benfordsches Gesetz da:Benfords lov it:Legge di Benford he:חוק בנפורד th:กฎของเบนฟอร์ด