I tried to design the ui with some ui mocking software, but i found it's hard for me to settle down all the detail of the design, since the database didn't design yet.
But if i first design software, then the same problem occur, I didn't have the UI, how can I create a prominent UI ?
usually you need to manipulate some data in the solutions you develop. So you should consider how this data is organised first, stabilizing this layer is fundamental at the beginning. I agree with duffymo's comment about designing the business objects first if you are in a OO world. Mapping these objects to the DB will be a part of your work. Then you add business functionality and work on the presentation layer. Of course you will have to do some refactoring from time to time, but usually the refactoring impacts the business and presentation layers more than the database.
read this, it is a good technique. DDD - Domain Driver Design
All the above answers address your issue in a right direction. That said, I would follow the SDLC thoroughly. It helps you understand the need for the solution for the problem at hand. Then comes the requirement gathering followed by the design either UI / underlying structure that supports the UI. It's a procedure but you would benefit in the end.