Skip to content

Fix class Dms double to float rounding

Fabrizio Pollastri requested to merge fabrizio/kstars:dms_double_float into master

Overview

In class Dms, all methods extracting the decimal part of the angle value in formats (DDD),MM,SS,mSS or (HH),MM,SS,mSS suffers from a rounding error that occurs, under certain circumstances, in the explicitly requested conversion from double to float. The methods are arcmin, arcsec, marcsec, minute, second, msecond.

All these methods convert a double precision expression to a float. When the angle to be converted is just below an integer value by an amount less than the float precision (1.0/(2e24)=5.960464477539063e-08), the float conversion rounds up to the unit the decimal part of the angle value. For example, let the angle be 10.0 degrees, after some trigonometric computation it becomes 9.99999999991, in this case, arcmin returns 60' instead of 59'. So, the toDMSString method will return 9°60'00" instead of 9°59'59".

The fix simply remove the float conversion.

A specific test is also added.

Details

  • The fix of class Dms is in .../kstars/auxiliary/dms.cpp
  • The specific test is added to .../Tests/auxiliary/testdms.cpp and .../Tests/auxiliary/testdms.h

Merge request reports