Surface Book Discussion Thread

Discussion in 'Microsoft' started by DRTigerlilly, Oct 6, 2015.

  1. ATIVQ

    ATIVQ V⅁O⅄ Senior Member

    Messages:
    1,043
    Likes Received:
    588
    Trophy Points:
    131
    ...but all the info is disclosed. Geekbench tells you exactly which algorithms it tests. SHA1 is ubiquitous, and it just happens that Apple uses ARM processors that are really good at multicore SHA1. No one is fudging the numbers.

    I don't even know what's your point - should I tell people to download 2.4GBs of software just to see if their device is performing as expected?
     
  2. Mesosphere

    Mesosphere Geek. Senior Member

    Messages:
    5,313
    Likes Received:
    2,060
    Trophy Points:
    331
    I disagree with your premise here. First and foremost people use benchmarks to compare different devices, and that is exactly where Geekbench seems to fall flat (at least when comparing windows/x86 to iOS/Arm machines). Specifically this was brought up with reference to the iPad Pro and Apple's claims about A9X's performance relative to much higher TDP iCore chips.

    However, I agree Sonic's suggested benchmarking software is also not the solution, since (among other reasons you pointed out) it wont run on an iOS, so it can't be used to compare either.

    In reality I think every benchmark is necessarily skewed. System "speed" is an inherently multi-dimensional thing, so any time you try collapse it to say something like system A is X% faster than system B it is always specific to a specific use case or an arbitrary weighting of a variety of use cases (most benchmarks fall in the latter category).
     
  3. freya301

    freya301 Pen Pal - Newbie

    Messages:
    69
    Likes Received:
    23
    Trophy Points:
    16
    Well, since apple isn't even an option for me, I tend to use geekbench to compare between various intel chips, which presumably avoids this bias. I do agree with dstrauss though, reading the two number score (single-core & multi-core) will be an oversimplification of the truth. Then again, I always scroll down on a particular geekbench benchmark and look at the exact measurements, with units, for whatever it is that I care about at the moment. But, on the other other hand, even this isn't the full picture, because of course one of the main benefits of skylake over haswell is the new hardware encoding, which isn't measured at all on this benchmark, nor are any potentially GPU-accelerated tests, a huge factor when assessing if a machine will be suitable for one's needs, nor is it clear how much snappier everything might feel with that sweet new PCIe SSD...

    I guess what I'm saying is that simplifications are a bad idea! Ditch benchmarks and absorb the hive-mind-impression of the device by reading 63 pages of thoughts in this handy forum! :) :)

    IMHO, anything less risks spending money on a device that might not live up to expectations.
     
    ATIVQ likes this.
  4. Marty

    Marty Pen Pro - Senior Member Senior Member

    Messages:
    3,493
    Likes Received:
    3,462
    Trophy Points:
    231
    I'm pretty sure you already looked at the linked tables in my previous post, so I won't berate the point. I'll simply say that there is no reasonable multi-threaded algorithm that an i7-2820QM could lose to an A9 at crunching out a SHA1.

    The basic argument here is that geekbench's processors scores do not scale linearly for sustained high-clock speed, multi-core architectures. Therefore, if they are comparing across cpu architectures, they aren't getting anything close to "rough estimate of their device's performance".

    This is the danger of using geekbench as the "layman's measure" of cpu performance--it fuels misconceptions which get repeated in headlines/social media.

    Never trust the hive mind (we make mistakes too). :p

    Nothing beats your own research and testing. ;)
     
    Last edited: Oct 21, 2015
  5. Mesosphere

    Mesosphere Geek. Senior Member

    Messages:
    5,313
    Likes Received:
    2,060
    Trophy Points:
    331
    My understanding is that it isn't about an algorithm, but rather hardware acceleration vs. software. The A9X has hardware SHA1 instructions that give it a significant edge over a software implementation necessary on an x86 chip.
     
  6. Steve S

    Steve S Pen Pro - Senior Member Super Moderator

    Messages:
    8,329
    Likes Received:
    3,841
    Trophy Points:
    331
    I think one thing that might be getting overlooked in this discussion is that benchmark scores are frequently mentioned without any qualifiers in discussion threads and that "layman" buyers simply take them at face value, never realizing that there are, in fact, significant caveats to the scores. In this scenario, a buyer could very well look at a Dell score of 3400 versus an Apple score of 1800 and conclude that he really needs to buy the Dell.

    Except for this consideration, and although I'm on record as being critical of the relevance of benchmarking, I don't lean toward either side of this argument; both camps have made good points...
     
    czm2000 likes this.
  7. Marty

    Marty Pen Pro - Senior Member Senior Member

    Messages:
    3,493
    Likes Received:
    3,462
    Trophy Points:
    231
    If you're referring to the hardware AES encryption instruction set, I believe the SandyBridge architecture also had that. Considering this and the fact that the 2820QM has a base frequency of 2.3-3.4GHz (vs A9's 0.8-2GHz) and two more physical cores, I still find the result to be very unlikely.

    My basic point was there's more testing going on than simply a SHA1, but whatever that is, geekbench has shown it does not scale well to sustained high-frequency, multi-core architectures.

    On the contrary, in my opinion, the whole idea of a benchmark score is to reduce performance to a form easily digestible to a layman. This is often in the form a single number, which represents the average of many specific contexts (eg. Javascript/Video Decoding/3D Rendering).

    It is the job of the benchmark to ensure this number scales appropriately between different architectures. For example, you never see the averaged 3DMark score of a high wattage GTX graphics card come up slightly less than a mobile gpu.

    But that is exactly the kind of distorted perspective that geekbench is promoting with its processor scores. In this situation, I believe "taken with caveats" is not enough--the result should be discarded in any realistic performance comparison.
     
    Last edited: Oct 21, 2015
  8. ATIVQ

    ATIVQ V⅁O⅄ Senior Member

    Messages:
    1,043
    Likes Received:
    588
    Trophy Points:
    131
    If it's so unlikely then just run SHA1 a million times on an i7-2820QM and on an ARM A7 and compare the results. It's not like Geekbench uses a black box that spits out numbers - it tells you what algorithms it's testing, and they're defined by open standards in the case of SHA1.

    For the fun of it let's compare Geekbench 3 scores with BAPCo TabletMark V3 scores, which was suggested earlier:

    Code:
    PROCESSOR      TABLETMARK3    GEEKBENCH3   
    --------------------------------------------
    i7-3770S       3218           3740
    i7-4650U       2547           3313
    i5-3210M       2262           2996
    M 5Y70         2423           2567
    M 5Y10         1873           2187
    Z3580          864            917
    TableMark v3 gives a Core M 5Y70 a higher score than a Core i5 3210M... And it requires a 2.4GB download.... and there are ten different software suites to download so you may end up with the wrong one... and they don't specify what algorithms they use... and they don't run on iOS... and they have a very small sample size so it's harder to compare...

    So yeah, I'm sticking to Geekbench.
     
  9. FoX4305

    FoX4305 Pen Pal - Newbie

    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    5
    yeah ! :) Surface Book shipped and estimate delivery for Friday !
     
  10. Marty

    Marty Pen Pro - Senior Member Senior Member

    Messages:
    3,493
    Likes Received:
    3,462
    Trophy Points:
    231
    Unfortunately, I don't happen to have those processors on hand at the moment. But if I could find evidence that the i7-2820QM is significantly faster, would that help convince you that the benchmark is flawed?

    Looking at your table, don't you think the tabletmark scores gives a much better indicator of the relative performance landscape? The example you cited between the Core M 5Y70 and Core i5 3210M is off; that is why I recommend looking at more than one benchmark.

    For example, I actually recommend Passmark and NotebookCheck as the best "quick reference" databases for x86 benchmark comparisons; they tell you the sample size and notebookcheck gives a very useful median range graph.

    However, it's true that those databases don't reveal the exact algorithm. Could you link to a whitepaper of Geekbench's methodology? I'm actually quite curious as to how it is arriving at those numbers.
     
Loading...

Share This Page