What Do You Think About - Reviews - Research - Testing - Subjective Viewpoints - Philosophy
Posted by: Barry Cleave in Articles, tags: Articles, Freeware, My Free Review, Portable Freeware, Portable Freeware Portable Application, PortableApps, Programs, ProReviewer, Software, Utilities, Windows, Windows XPWarning: This is a subjective and slightly philosophical post - My Personal Viewpoint - Feel Free to Disagree or Just Ignore
What resources does a program use, how do you measure them, does it mean anything?
Research, software testing, subjective viewpoints of the user all present challenges to giving expert opinion.
I received an interesting and valuable comment on my AVG 8 Free post (NEW - AVG Anti Virus 8 FREE - Quick Review) from mulligrub:
Good review but a few questions.
The 2 biggest pluses for AVG free in the past are:
1. Free &;
2. Very low overhead - doesn’t use much system resources in day to day operation
The negatives include:
Not the fastest when scanning
Only daily upadtes
Outdated User Interface (looks like that has been addressed)
No Spyware, rootkit, spam protection (Spyware included in AVG8)
So if we are getting more (spyware scanning) in a flashier package
still for free the only achillies heel I can see will be the resource
usagehave you compared what level of resources it is using now?
Does the user experience ‘feel’ slower?
I am keen to know before recommending to family and friends and will test for myself too
Cheers
Well firstly thanks to mulligrub for the comment, and secondly thanks for sparking my interest to write this post!
I felt a more detailed answer and explanation is required, far more than just a comment reply on the original post. This is because it gets into some of the basic philosophy and policy I have tried to put in place for this blog.
A bit of background information is relevant to explain where I am coming from:
When I first decided to start this blog I was keen that it should provide information in a way that was a bit different to some of the other blogs in this area. I wanted to give readers a bit more detail in order to help them with decision making in terms of whether it was worth their time and effort to try a particular program or utility. The idea was to help with the decision making process not to make the decision and say this is “best” or don’t use something because it performs badly in some way or another. That I want to write about something, a program, a tip or some advice, means that I feel it is worth some effort and will be useful to the reader.
I was also keen not to get into the position of being overly judgmental on the merits of any particular item of free software against another. In particular I was very aware that I would never have the time to carry out proper comparative testing, so any opinion I could ever give would be subjective and based on a very few available facts.
It is my background as an engineer and researcher/scientist that makes me cautious to provide opinion without proper testing. I am more comfortable with giving my personal and very subjective viewpoint. The reader can value that, or not, based on their opinion of my posts.
To understand where I am coming from a little more information. I am a Civil Engineer by profession and my work over the past 10 years has involved me in research and “high level” consultancy into the performance of infrastructure assets (highway assets - that is anything that is built as part of the highway infrastructure - roads, bridges, tunnels, drainage, lighting, signs, barriers, signaling systems, etc.) and into the development of systems to manage them which includes developing software tools. Therefore, I hope I am in position to talk with some expertise on research/testing and subjective views/perception.
My approach is therefore from this point of view when asked particular questions such as:
- Have you compared what level of resources it is using now?
- Does the user experience ‘feel’ slower?
Question 1 raises several points regarding research/testing:
To “scientifically” measure resources to compare the two versions of AVG (or any two or more software utilities) would require defining a detailed test procedure that was, as far as possible, exactly the same for both versions. It would also require to properly test all the various working aspects of the software over a range of conditions, hardware and operating environments, some of which may use more or less of the defined resources to be evaluated than the other, this would include -
What resources are important - What difference would a limit on one or more resource in the test set up make to the results eg slow CPU, small amount of RAM, slow FSB, slow disk drives, etc.
I could raise a lot of other “variables” that could effect the results and opinion but I hope you can see the difficulty.
Furthermore from the ProReviewer point of view, I only currently have the ability to test under an XP OS environment on a couple of PCs, so I wouldn’t be able to do justice to the software I was reviewing at the level of detail required to carry out proper detailed comparative testing, even if I had the time.
Many of the professional PC magazines cover this area fairly well but even with these you can see there has to be limits to their testing methods. They can’t test all possible combinations of software, hardware and operating systems.
Question 2 is also very interesting because it raises the question of subjectivity:
Here I am happy to give MY subjective viewpoint, this is what I do in the “What is the Proreviewer Experience” section of my review posts. What you must understand is that subjective means that is how I perceive the software.
Again I feel I need to write a quick note on subjectivity.
If you open a program and it is up and running in a couple of seconds you are probably happy. If it takes a minute you won’t be. Or is that really true, if it takes a minute but it is an autostart program do you really notice? Maybe yes - maybe no.
If you run your PC 24/7 and have an Antivirus program set to scan your system at 2.00am in the morning when you are asleep until 6.00am do you care if it takes 4 hours. If you only scan your system in the brief periods you turn your computer on and it slows everything down then obviously you will care a lot!
If your Antivirus program “realtime” scanning of files (as they are opened by an application) has an impact of say 0.1 second when a file opens do you care? Probably not.
However if you have say desktop search software scanning thousands of files for content then the 0.1 second per file has a very real impact on system performance. Now you really do care.
Have you noticed how when you need to open a document and print it, or go to a website for some directions to a place when you are already late leaving seems to take forever?
I hope you can see that your subjective view of a program is dependent on many factors but principally how it effects your experience and what you are trying to accomplish.
Congratulations to anyone who has stayed awake long enough to read this far into the post
.
Feel free to tell me what you think just remember:
A couple of quotes to put things in context:
There are no stupid questions - just stupid answers!
——————————————————————–
A man is getting along on the road to
wisdom when he begins to realize
that his opinion is just an opinion.
——————————————————————–
One accurate measurement is worth
a thousand expert opinions.
——————————————————————–
What Other Relevant Posts are There?
NEW - AVG Anti Virus 8 FREE - Quick Review
Save Time, Effort & Maybe Money - Use Web Reviews Before you Install or Purchase - Part 1
Save Time, Effort & Maybe Money - Use Web Reviews Before you Install or Purchase - Part 2
What are the Tags?
Popularity: 41% [?]











Entries (RSS)
May 1st, 2008 at 1:45 am - Edit
Hi again
I think your ‘manifesto’ above is great and it is what keeps me reading your posts
Your articles give facts and then opinions separately which is a good way to do it
From my point of view so long as we can identify what are hard facts and what are subjective opinions etc then we are fine
Readers identify with you and your opinions and that’s what bring them back so don’t be afraid to offer them
As you say the ‘big boys’ can do more thorough documenting of the hard facts and empirical testing but I suspect readers of your blog aren’t looking for that - they just want some facts to lay a basis for discussion of your experiences
I think a comment like “on my system AVG7 used ?? resources and AVG8 uses ??” or “I haven’t noticed any difference in start up time or scanning time” in your Experience section doesn’t invalidate your post - more it explains how you formed your opinion.
Anyway keep up the good work mate. I promise to keep reading
May 1st, 2008 at 8:05 pm - Edit
Hi Mulligrub,
Thanks for the comment, thanks for the email and thanks again for the encouragement
I’ll certainly have a look at your site soon (only delayed by the fact I am off for a long weekend to see my daughter and grandchildren).
I certainly try to give some exact figures where possible but I am always cautious as things like memory use will depend on what the software is doing at the time. The other problem I have at the moment is that my current XP installation on my primary PC is grinding to a halt and needs to be reinstalled (rolled back to a cleaner backup) so subjective views are completely compromised. Anyway the reinstall will give me some opportunities for posts so it is no bad thing.
All the best
Barry Cleave
May 23rd, 2008 at 10:16 am - Edit
I know how you feel; let me share some of my experience with software tinkering.
First of all, I never trust magazines - in my opinion, their reviews are supersuperficial; and quite often they are ads disguised as interesting articles. The other thing about these magazines is that somehow they always happen to have a test configuration which exceeds the horse-power of my PC; their figures will not be relevant in my case. At first I thought it was my being conservative and not upgrading often, but my personal statistics bank shows me that this applies to many other people. Most of the world does not use a NASA server as their home desktop, period.
Magazine writers test their stuff on bleeding-edge hardware given to them by hardware manufacturers, seeking bonus points. So what we have is a biased article about a program tested on modern hardware which only a small subset of the population can afford. As a mere mortal, I can’t keep up with them.
In other words, magazines are not a trustworthy source of information. I prefer Internet forums or blogs, where content is written by “real humans”.
So, how do I tell a good review from a bad review? It must not be magazine-like. When I see that a real human wrote it, I know I’m dealing with someone’s personal opinion, and not someone’s implementation of “the opinion that has to be expressed”. Of course you won’t cover all the aspects I am interested in, but what you offer will be used as a piece in the bigger puzzle; I will then look for other reviews and build the big picture myself.
My computer/software usage patterns are quite uncommon, so there is no “one review to rule them all”.
Criteria that I use to evaluate a program [incomplete list]:
- how much memory it uses when doing different tasks (ex: audio player - idle, playing)
- does it create any startup entries
- is there a tray icon (if yes, can I disable it if I don’t need it)
- does it use native GUI widgets or not (non-native widgets usually mean that the program consumes a lot of memory just for eye-candy)
- does it conform, or try to conform, with the HIG (human interface guidelines) that are specific to the OS?
- does it “phone home”?
- what is the startup time? (I am willing to wait if it is a mail client I load once and never close afterwards, but if it is a small tool, my patience is limited )
- does it create any file associations, or modify existing ones without asking?
- how buggy is it?
Of course, there are always exceptions. For example Trillian uses its own widgets; Inkscape doesn’t look like a native Windows application - but it follows the Gnome HIG, looks good, works well; Picasa doesn’t use native widgets either - but I can’t find anything better than that, I use it rarely so I am willing to pay the price
Therefore, one can never learn everything from a single review, too many ad-hoc decisions have to be made.
Alex Raileans last blog post..To love, or not to love?
May 23rd, 2008 at 11:45 am - Edit
Hi Alex,
Great comment, a post all on its own!
I’ll take on board a copuple of your points for future reviews.
Barry
June 2nd, 2008 at 2:48 am - Edit
[...] [...]
August 6th, 2008 at 6:02 am - Edit
[...] What Do You Think About - Reviews - Research - Testing - Subjective Viewpoints - Philosophy [...]
August 6th, 2008 at 2:19 pm - Edit
[...] XP OS is Dying (and Holiday Weekend in the UK) | Pro Reviewer (My Free Review) (0 comments.) on What Do You Think About - Reviews - Research - Testing - Subjective Viewpoints - PhilosophyTip - Help Avoid Duplicate Emails in MS Outlook - Change Timeout Settings | Pro Reviewer (My Free [...]