I tried to design the ui with some ui mocking software, but i found it's hard for me to settle down all the detail of the design, since the database didn't design yet.
But if i first design software, then the same problem occur, I didn't have the UI, how can I create a prominent UI ?
Put the user at the place he deserves. Design UI first.
Database is only a consequence of user needs.
use cases first, neither ui nor database.
Would you build a house without a foundation? Database design isn't the fun part but it is the foundation of most business apps and if you get it wrong it becomes the most costly to fix and the most costly to maintain.
That said, I note that there is no reason you can't work on both together as they intertwine. But before you can do either, you need to understand the requirements and the business you are writing the app for.
If you're trying to solve a problem in an object-oriented language, it's recommended that you start thinking about the objects involved. Don't worry about the database or the UI until you've got a solid domain model nailed down that addresses all the use cases.
You don't worry about the database or the UI at first. You can serialize objects to the file system if you need persistence and don't have a database. Being able to drive your app with a command line UI is a good exercise for guaranteeing that you have a good MVC separation.
Start with the objects.
UPDATE:
The one advantage that this approach has is that it doesn't prejudice the UI with a particular database design or vice versa. The object are agnostic about the other two layers. You aren't required to have a UI or relational database at all. You're just getting the objects right. Once you have that, you can create any UI or persistence scheme you like, confident that the domain model can handle the problem you've been asked to solve.
Your question is very subjective.
My opinion (and it is just that) is that database and underlying structure should come first. It can often help to put down the keyboard and mouse and compile some notes on paper.
Lay out goals like what you want your application to do, list the features you require and then start thinking about how you'll build it.
This method works for me in application design.
UI first.
Mock up an elegant and easy-to-use user interface (and workflow) thinking from the point of view of the user, and only then think about the underlying database / data structures you'll need to bring that UI to life.
If you can't design your UI because you haven't yet designed your database, you're doing it wrong IMHO. How many annoying pieces of software have you used that suffered from letting the database design drive the UI design?
Edit: As others have pointed out, you need to start with use cases / user stories. The UI design and database design, whichever order you do them, should only happen after you know what your software is trying to do, and for whom.
Edit by Bryan Oakley:
it's not what the software does. it's what the user does http://www.gapingvoid.com/114446615687.jpg