Yahoo Groups archive

Lpc2000

Index last updated: 2026-04-28 23:31 UTC

Thread

On KEILs Dhrystone benchmarks --again

On KEILs Dhrystone benchmarks --again

2005-05-04 by ukkie9a

Hello everyone,

To satisfy my curiosity (and verify the code generation of the GNU
compiler), I repeated the Dhrystone benchmark with GNU GCC after
reading the tables in the (otherwise nice book) "The Insider's Guide
to the Philips ARM7-Based Microcontrollers" by Trevor Martin (see
http://www.hitex.co.uk/arm/lpc2000book/index.html) and finding the
same table on KEILs web site.

My results, which you can read about on
http://compuphase.com/dhrystone.htm, differ from the published results
by a factor of 5!

Kind regards,
Thiadmer Riemersma

Re: [lpc2000] On KEILs Dhrystone benchmarks --again

2005-05-05 by Rod Moffitt

Great work Thiadmer!

I personally ignore benchmarks because most of the time the results are 
skewed one way or another, and rarely are the test parameters and raw 
results publicized.

If anyone from Keil is lurking on this list I would highly recommend 
responding. It either appears that your techs borked the tests or used 
some sort of 'invalid' test setup. In either case I recommend re-running 
the tests, and this time providing the raw tests results. In case you made 
a mistake I would suggest an apology on your web site.

On a side note, I use GNU tools not because of the speed/power (either for
or against) however because of the openness of the source. I have used
many closed-source tools and each vendor has let me down at least once
with support (no matter what large or small company I worked for). With 
GNU I can always take a look at the source in case of a problem. And (more 
importantly!) I can choose WHATEVER OS I want for development thanks to 
having source for all the tools.

On another side note: at work I work with QNX, and guess what their 
compiler is? It's referred to as 'qcc' yet in fact it is GCC (2.95.3 and 
3.3 at the moment). Many other closed source vendors in fact use GCC, yet 
don't necessarily publicize the fact. So if anyone ever thinks GCC is not 
'good enough' for real embedded development, think again!

- Rod

--
                          ___  ____  ___    _      ___
  Rod Moffitt            / _ \/ __ \/ _ \  (_)__  / _/__
  http://rod.info       / , _/ /_/ / // / / / _ \/ _/ _ \
  rodANTISPAM@... /_/|_|\____/____(*)_/_//_/_/ \___/
  =======================================================
  ~ Where loved ones are remembered http://memoriam.org ~
Show quoted textHide quoted text
On Wed, 4 May 2005, ukkie9a wrote:

> Hello everyone,
>
> To satisfy my curiosity (and verify the code generation of the GNU
> compiler), I repeated the Dhrystone benchmark with GNU GCC after
> reading the tables in the (otherwise nice book) "The Insider's Guide
> to the Philips ARM7-Based Microcontrollers" by Trevor Martin (see
> http://www.hitex.co.uk/arm/lpc2000book/index.html) and finding the
> same table on KEILs web site.
>
> My results, which you can read about on
> http://compuphase.com/dhrystone.htm, differ from the published results
> by a factor of 5!
>
> Kind regards,
> Thiadmer Riemersma
>
>
>
>
>
> Yahoo! Groups Links
>
>
>
>
>
>
>

Re: [lpc2000] On KEILs Dhrystone benchmarks --again

2005-05-05 by Leon Heller

----- Original Message ----- 
Show quoted textHide quoted text
From: "Rod Moffitt" <rodlist@...>
To: <lpc2000@yahoogroups.com>
Sent: Thursday, May 05, 2005 3:25 AM
Subject: Re: [lpc2000] On KEILs Dhrystone benchmarks --again


> Great work Thiadmer!
>
> I personally ignore benchmarks because most of the time the results are
> skewed one way or another, and rarely are the test parameters and raw
> results publicized.
>
> If anyone from Keil is lurking on this list I would highly recommend
> responding. It either appears that your techs borked the tests or used
> some sort of 'invalid' test setup. In either case I recommend re-running
> the tests, and this time providing the raw tests results. In case you made
> a mistake I would suggest an apology on your web site.
>
> On a side note, I use GNU tools not because of the speed/power (either for
> or against) however because of the openness of the source. I have used
> many closed-source tools and each vendor has let me down at least once
> with support (no matter what large or small company I worked for). With
> GNU I can always take a look at the source in case of a problem. And (more
> importantly!) I can choose WHATEVER OS I want for development thanks to
> having source for all the tools.
>
> On another side note: at work I work with QNX, and guess what their
> compiler is? It's referred to as 'qcc' yet in fact it is GCC (2.95.3 and
> 3.3 at the moment). Many other closed source vendors in fact use GCC, yet
> don't necessarily publicize the fact. So if anyone ever thinks GCC is not
> 'good enough' for real embedded development, think again!

Microchip uses gcc in the C30 compiler for their new dsPIC devices - it 
works very well. As with most DSP compilers, the libraries are written in 
assembler.

Leon
--
Leon Heller, G1HSM
http://www.geocities.com/leon_heller

Re: On KEILs Dhrystone benchmarks --again

2005-05-05 by Richard

Yes, I hope that Keil responds as well.  If for no other reason than
to admit to failing to turn on the MAM or to provide other reasons for
these discrepancies.

--- In lpc2000@yahoogroups.com, Rod Moffitt <rodlist@r...> wrote:
> Great work Thiadmer!
> 
> I personally ignore benchmarks because most of the time the results are 
> skewed one way or another, and rarely are the test parameters and raw 
> results publicized.
> 
> If anyone from Keil is lurking on this list I would highly recommend 
> responding. It either appears that your techs borked the tests or used 
> some sort of 'invalid' test setup. In either case I recommend
re-running 
> the tests, and this time providing the raw tests results. In case
you made 
> a mistake I would suggest an apology on your web site.
> 
> On a side note, I use GNU tools not because of the speed/power
(either for
> or against) however because of the openness of the source. I have used
> many closed-source tools and each vendor has let me down at least once
> with support (no matter what large or small company I worked for). With 
> GNU I can always take a look at the source in case of a problem. And
(more 
> importantly!) I can choose WHATEVER OS I want for development thanks to 
> having source for all the tools.
> 
> On another side note: at work I work with QNX, and guess what their 
> compiler is? It's referred to as 'qcc' yet in fact it is GCC (2.95.3
and 
> 3.3 at the moment). Many other closed source vendors in fact use
GCC, yet 
> don't necessarily publicize the fact. So if anyone ever thinks GCC
is not 
Show quoted textHide quoted text
> 'good enough' for real embedded development, think again!
> 
> - Rod
> 
> --
>                           ___  ____  ___    _      ___
>   Rod Moffitt            / _ \/ __ \/ _ \  (_)__  / _/__
>   http://rod.info       / , _/ /_/ / // / / / _ \/ _/ _ \
>   rodANTISPAM@r... /_/|_|\____/____(*)_/_//_/_/ \___/
>   =======================================================
>   ~ Where loved ones are remembered http://memoriam.org ~
> 
> On Wed, 4 May 2005, ukkie9a wrote:
> 
> > Hello everyone,
> >
> > To satisfy my curiosity (and verify the code generation of the GNU
> > compiler), I repeated the Dhrystone benchmark with GNU GCC after
> > reading the tables in the (otherwise nice book) "The Insider's Guide
> > to the Philips ARM7-Based Microcontrollers" by Trevor Martin (see
> > http://www.hitex.co.uk/arm/lpc2000book/index.html) and finding the
> > same table on KEILs web site.
> >
> > My results, which you can read about on
> > http://compuphase.com/dhrystone.htm, differ from the published results
> > by a factor of 5!
> >
> > Kind regards,
> > Thiadmer Riemersma
> >
> >
> >
> >
> >
> > Yahoo! Groups Links
> >
> >
> >
> >
> >
> >
> >

Re: On KEILs Dhrystone benchmarks --again

2005-05-05 by roger_lynx

Hello everybody,

I applaud the critical reasoning and writing style used in the
preparation of the paper referenced below. 
There are two issues, that I'd like to raise, that are by no means
intended as to discount the quality of the paper. 
However:

A. The KEIL benchmark used LPC2294 in their *simulator* to obtain
those published values.
B. Thiadmer's paper describes benchmarking of a *real* LPC2106 system.

To acknowledge the difference, and then to proceed with the testing -
puzzles me (see quote below).

The means and ways of the benchmark code compilation [unreferenced in
KEIL source] under respective compilers could skew only the results
for the size of the executables. But the size of the code is only one
aspect of the cited benchmarks [this is an area where Thiadmer's paper
excels].
The other aspect is the execution speed in two different environments,
that I view as uncomparable. 

Is the objective to benchmark the CPU simulator or the real HW?
Are those two identical?

There is probably NO better compiler than a sharp mind, fortified by
large amount of experience, an assembler, infinite time and financial
resources. 
In the land of Shan-gri-la.

'nuff said.

--roger

Edited quote:
" My controller is close enough to the test setup of KEIL
(µVision Simulator that emulates a Philips LPC2294 running at 60 MHz
in Thumb Mode). Still, my benchmarks run significantly faster than
what KEIL documents, and the sizes of my programs are smaller than any
of the benchmark programs from KEIL. 
It is obvious that, despite the similarity in the specifications of
our benchmarks, KEIL and I have tested entirely different systems."

--- In lpc2000@yahoogroups.com, "ukkie9a" <docent@c...> wrote:
Show quoted textHide quoted text
> Hello everyone,
> 
> To satisfy my curiosity (and verify the code generation of the GNU
> compiler), I repeated the Dhrystone benchmark with GNU GCC after
> reading the tables in the (otherwise nice book) "The Insider's Guide
> to the Philips ARM7-Based Microcontrollers" by Trevor Martin (see
> http://www.hitex.co.uk/arm/lpc2000book/index.html) and finding the
> same table on KEILs web site.
> 
> My results, which you can read about on
> http://compuphase.com/dhrystone.htm, differ from the published results
> by a factor of 5!
> 
> Kind regards,
> Thiadmer Riemersma

Re: On KEILs Dhrystone benchmarks --again

2005-05-05 by berniespam

I do not think Keil was trying to be insidious with their comparison
of their tools with other ARM compilers. If so, then Keil would be #1
in all reported parameters. 

Using a widely recognized benchmark like Dhrystone removes the chance
that an the benchmark is written to favor the quirks of a particular
compiler. 

They include the source code that they used to run the test on their
evaluation CD.

The GNU source code for the Dhrystone is on the CD-ROM at:
  \Keil\ARM\GNU\Examples\DHRY\

The Keil source code is the Dhrystone on the CD-ROM at:
  \Keil\ARM\Examples\DHRY\

Using the simulator allowed the test to be hardware neutral. Since the
simulator will take a hex file created by any tool set, it should not
favor any particular compiler. It also makes it easier for an end user
to reproduce the test.

No benchmark is perfect; just because a compiler does well under one
benchmark, does not mean your source code will do equally well. 

--- In lpc2000@yahoogroups.com, "ukkie9a" <docent@c...> wrote:
Show quoted textHide quoted text
> Hello everyone,
> 
> To satisfy my curiosity (and verify the code generation of the GNU
> compiler), I repeated the Dhrystone benchmark with GNU GCC after
> reading the tables in the (otherwise nice book) "The Insider's Guide
> to the Philips ARM7-Based Microcontrollers" by Trevor Martin (see
> http://www.hitex.co.uk/arm/lpc2000book/index.html) and finding the
> same table on KEILs web site.
> 
> My results, which you can read about on
> http://compuphase.com/dhrystone.htm, differ from the published results
> by a factor of 5!
> 
> Kind regards,
> Thiadmer Riemersma

Re: On KEILs Dhrystone benchmarks --again

2005-05-07 by Thiadmer Riemersma (ITB CompuPhase)

Hello "berniespam",

> I do not think Keil was trying to be insidious with their comparison
> of their tools with other ARM compilers.

I have tried to choose my words carfully, to avoid accusing KEIL of
anything. (Except that I do feel that KEIL should have provided an
analysis.)

> Using a widely recognized benchmark like Dhrystone removes the
> chance that an the benchmark is written to favor the quirks of a
> particular compiler.

I think you were referring to the remark that Dhrystone may be
inappropriate for embedded systems software. For that remark, I cited
Richard York, and a link to his paper is near the bottom of my
article.

> Using the simulator allowed the test to be hardware neutral. Since
> the simulator will take a hex file created by any tool set, it
> should not favor any particular compiler. It also makes it easier
> for an end user to reproduce the test.

This is a good point. I will take this into account when I update my
article on the Dhrystone benchmark.

Thanks for posting, anf kind regards,
Thiadmer Riemersma

Re: On KEILs Dhrystone benchmarks --again

2005-05-07 by Thiadmer Riemersma (ITB CompuPhase)

Hello Rod Moffitt,

> I personally ignore benchmarks because most of the time the results
> are skewed one way or another, and rarely are the test parameters
> and raw results publicized.

That said, "benchmarks followed by analysis" can improve your
understanding of the system.

For example, I have been developing software for a new product on a
test board containing an LPC2106, but the actual apparatus will
contain an LPC2138. One of the first things I will do on the prototype
of the apparatus (which I will have access to Monday or Tuesday) is to
run Dhrystone, to check whether I have set up the chip correctly. Now
that I know what to expect, any deviations will point me in the right
direction.

> If anyone from Keil is lurking on this list I would highly
> recommend responding. It either appears that your techs borked the
> tests or used some sort of 'invalid' test setup.

And this may very well have been unintentional. I have made all of the
"stupid mistakes" in benchmarking throughout my career. That is
another reason why benchmark results must be analysed. A simple trick
is to browse through the assembler code that the compiler generates.

Kind regards,
Thiadmer

Re: On KEILs Dhrystone benchmarks --again

2005-05-07 by Thiadmer Riemersma (ITB CompuPhase)

Hello Roger,

> A. The KEIL benchmark used LPC2294 in their *simulator* to obtain
> those published values.
> B. Thiadmer's paper describes benchmarking of a *real* LPC2106
> system.
> 
> To acknowledge the difference, and then to proceed with the testing
> - puzzles me (see quote below).

Good point. Based on the claim that the KEIL simulator is "cycle
accurate" and the LPC2294 and LPC2106 using the same ARM7TDMI core, I
*assumed* the results would be comparable. Obviously, I should have
verified my assumptions. I intend to update my article with any new
insight (but I must ask for some patience...).

Thank you for your comment, and kind regards,
Thiadmer

Re: [lpc2000] On KEILs Dhrystone benchmarks --again

2005-05-09 by Michael Anburaj

Hi Thiadmer,

I am curious about your results.

I ran Dhrystone 2.1 on LPC2106 board a year ago.

Results:
http://geocities.com/michaelanburaj/downloads/dhry_lpc.gif

I am heavy user of GCC tools (ARM, MIPS & other 32/64
bit RISCs). I run Dhrystone as apart of board bringup
each time I start with a new board/processor (always
with \ufffdO2 option) \ufffd same code. So, I always code a 1
second granularity timer function to make measurements
& there is timer sync. loop at the beginning. I make
measurements over large iterations to get accurate
results. I believe 10,000,000 iterations should give
fairly accurate results across different processor
platforms.

The only difference I see is I have used GCC 3.4.0 &
you used 3.4.3
Still 41 against 51 \ufffd in ARM mode & 35 against 44 \ufffd in
Thumb mode is huge.

Moving to 3.4.3 give better performance?

Thanks,
-Mike.

--- ukkie9a <docent@...> wrote:
> Hello everyone,
> 
> To satisfy my curiosity (and verify the code
> generation of the GNU
> compiler), I repeated the Dhrystone benchmark with
> GNU GCC after
> reading the tables in the (otherwise nice book) "The
> Insider's Guide
> to the Philips ARM7-Based Microcontrollers" by
> Trevor Martin (see
> http://www.hitex.co.uk/arm/lpc2000book/index.html)
> and finding the
> same table on KEILs web site.
> 
> My results, which you can read about on
> http://compuphase.com/dhrystone.htm, differ from the
> published results
> by a factor of 5!
> 
> Kind regards,
> Thiadmer Riemersma
> 
> 
>


		
__________________________________ 
Yahoo! Mail Mobile 
Take Yahoo! Mail with you! Check email on your mobile phone. 
http://mobile.yahoo.com/learn/mail

Re: [lpc2000] On KEILs Dhrystone benchmarks --again

2005-05-09 by Michael Anburaj

sorry I mistook Dhrystone/seconds (on your results)
for DMIPS. My results are after dividing it by 1757.

Then it looks like I have seen better numbers using
GCC. And 49 DMIPS with SDT -- is close to 54 DMIPS
advertised by Philips. I remember verifying my results
against numbers advertise by ARM a year ago & it was
close enough.

Cheers,
-Mike.

--- Michael Anburaj <embeddedeng@...> wrote:
> Hi Thiadmer,
> 
> I am curious about your results.
> 
> I ran Dhrystone 2.1 on LPC2106 board a year ago.
> 
> Results:
>
http://geocities.com/michaelanburaj/downloads/dhry_lpc.gif
> 
> I am heavy user of GCC tools (ARM, MIPS & other
> 32/64
> bit RISCs). I run Dhrystone as apart of board
> bringup
> each time I start with a new board/processor (always
> with \ufffdO2 option) \ufffd same code. So, I always code a 1
> second granularity timer function to make
> measurements
> & there is timer sync. loop at the beginning. I make
> measurements over large iterations to get accurate
> results. I believe 10,000,000 iterations should give
> fairly accurate results across different processor
> platforms.
> 
> The only difference I see is I have used GCC 3.4.0 &
> you used 3.4.3
> Still 41 against 51 \ufffd in ARM mode & 35 against 44 \ufffd
> in
> Thumb mode is huge.
> 
> Moving to 3.4.3 give better performance?
> 
> Thanks,
> -Mike.
> 
> --- ukkie9a <docent@...> wrote:
> > Hello everyone,
> > 
> > To satisfy my curiosity (and verify the code
> > generation of the GNU
> > compiler), I repeated the Dhrystone benchmark with
> > GNU GCC after
> > reading the tables in the (otherwise nice book)
> "The
> > Insider's Guide
> > to the Philips ARM7-Based Microcontrollers" by
> > Trevor Martin (see
> > http://www.hitex.co.uk/arm/lpc2000book/index.html)
> > and finding the
> > same table on KEILs web site.
> > 
> > My results, which you can read about on
> > http://compuphase.com/dhrystone.htm, differ from
> the
> > published results
> > by a factor of 5!
> > 
> > Kind regards,
> > Thiadmer Riemersma
> > 
> > 
> >
> 
> 
> 		
> __________________________________ 
> Yahoo! Mail Mobile 
> Take Yahoo! Mail with you! Check email on your
> mobile phone. 
> http://mobile.yahoo.com/learn/mail 
> 
> 
>  
> Yahoo! Groups Links
> 
> 
>     lpc2000-unsubscribe@yahoogroups.com
> 
>  
> 
> 
> 

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

Re: On KEILs Dhrystone benchmarks --again

2005-07-19 by berniespam

Hello Thiadmer, 

I was doing some comparisons of different compilers (IAR, ARM ADS,
GNU, Keil...), and I wanted to use your well-done analysis
methodology. The paper was posted at: 

http://compuphase.com/dhrystone.htm

But it appears to be down. 

Can you republish the paper, attach it to this thread, or email it to
me? I wanted to ensure a complete and through analysis; one that all
group members would find useful.

Once I am done, I will post my results on the yahoo lpc2000 group,
(giving you credit where credit is due, of course). 

Thanks



--- In lpc2000@yahoogroups.com, "ukkie9a" <docent@c...> wrote:
Show quoted textHide quoted text
> Hello everyone,
> 
> To satisfy my curiosity (and verify the code generation of the GNU
> compiler), I repeated the Dhrystone benchmark with GNU GCC after
> reading the tables in the (otherwise nice book) "The Insider's Guide
> to the Philips ARM7-Based Microcontrollers" by Trevor Martin (see
> http://www.hitex.co.uk/arm/lpc2000book/index.html) and finding the
> same table on KEILs web site.
> 
> My results, which you can read about on
> http://compuphase.com/dhrystone.htm, differ from the published results
> by a factor of 5!
> 
> Kind regards,
> Thiadmer Riemersma

Re: On KEILs Dhrystone benchmarks --again

2005-07-26 by Thiadmer Riemersma (ITB CompuPhase)

Hello Bernie,

I have noticed that our site was temporarily down too. I do not know
why. It should be up again. One issue may be that I made an error in
the URL in my original post; it should be:

http://www.compuphase.com/dhrystone.htm

When the "www" is absent, DNS may need somewhat longer to find the
domain, and with the current load of the entire Internet, the DNS
lookup may time out. A retry then usually works (except under Windows
XP, which caches failed lookups as well as successfull lookups).

Good luck with your analysis. I am looking forward to reading it.
Thiadmer

Re: [lpc2000] Re: On KEILs Dhrystone benchmarks --again

2005-07-26 by Michael Johnson

We've published our Dhrystone/Whetstone numbers (and the project files) at

www.rowley.co.uk/arm/arm_bench.htm

I suspect the remarkable Keil Whestone performance can be explained by 
the "Treat Double as Float" option being checked (and greyed out) in 
their IDE. Can someone from Keil (or one of their customers) comment on 
this?

Regards
Michael
Show quoted textHide quoted text
>Hello Bernie,
>
>I have noticed that our site was temporarily down too. I do not know
>why. It should be up again. One issue may be that I made an error in
>the URL in my original post; it should be:
>
>http://www.compuphase.com/dhrystone.htm
>
>When the "www" is absent, DNS may need somewhat longer to find the
>domain, and with the current load of the entire Internet, the DNS
>lookup may time out. A retry then usually works (except under Windows
>XP, which caches failed lookups as well as successfull lookups).
>
>Good luck with your analysis. I am looking forward to reading it.
>Thiadmer
>
>
>
>
>
> 
>Yahoo! Groups Links
>
>
>
> 
>
>
>  
>

Move to quarantaine

This moves the raw source file on disk only. The archive index is not changed automatically, so you still need to run a manual refresh afterward.